patrick/zephyr
1
0
Fork 0
Code Issues Pull requests Actions 7 Packages Projects Releases Wiki Activity
zephyr/subsys/rtio/rtio_executor_simple.c

149 lines
3.5 KiB
C
Raw Normal View History

rtio: Real-Time Input/Output Stream A DMA friendly Stream API for zephyr. Based on ideas from io_uring and iio, a queue based API for I/O operations. Provides a pair of fixed length ringbuffer backed queues for submitting I/O requests and recieving I/O completions. The requests may be chained together to ensure the next operation does not start until the current one is complete. Requests target an abstract rtio_iodev which is expected to wrap all the hardware particulars of how to perform the operation. For example with a SPI bus device, a description of what a read, and write mean can be decided by the iodev wrapping a particular device hanging off of a SPI controller. The queue pair are submitted to an executor which may be a simple inplace looping executor done in the callers execution context (thread/stack) but other executors are expected. A threadpool executor might for example allow for concurrent request chains to execute in parallel. A DMA executor, in conjunction with DMA aware iodevs would allow for hardware offloading of operations going so far as to schedule with priority using hardware arbitration. Both the iodev and executor are definable by a particular SoC, meaning they can work in conjuction to perform IO operations using a particular DMA controller or methodology if desired. The application decides entirely how large the queues are, where the buffers to read/write come from (some executors may have particular demands!), and which executor to submit requests to. Signed-off-by: Tom Burdick <thomas.burdick@intel.com>
2019-06-26 17:17:18 +02:00
/*
* Copyright (c) 2022 Intel Corporation.
*
* SPDX-License-Identifier: Apache-2.0
*/
rtio: Add callback op Adds a callback op to RTIO enabling C logic to be interspersed with I/O operations. This is not safe from userspace but allowable in kernel space. Signed-off-by: Tom Burdick <thomas.burdick@intel.com>
2023-02-17 19:16:05 +01:00
#include "rtio_executor_common.h"
rtio: Real-Time Input/Output Stream A DMA friendly Stream API for zephyr. Based on ideas from io_uring and iio, a queue based API for I/O operations. Provides a pair of fixed length ringbuffer backed queues for submitting I/O requests and recieving I/O completions. The requests may be chained together to ensure the next operation does not start until the current one is complete. Requests target an abstract rtio_iodev which is expected to wrap all the hardware particulars of how to perform the operation. For example with a SPI bus device, a description of what a read, and write mean can be decided by the iodev wrapping a particular device hanging off of a SPI controller. The queue pair are submitted to an executor which may be a simple inplace looping executor done in the callers execution context (thread/stack) but other executors are expected. A threadpool executor might for example allow for concurrent request chains to execute in parallel. A DMA executor, in conjunction with DMA aware iodevs would allow for hardware offloading of operations going so far as to schedule with priority using hardware arbitration. Both the iodev and executor are definable by a particular SoC, meaning they can work in conjuction to perform IO operations using a particular DMA controller or methodology if desired. The application decides entirely how large the queues are, where the buffers to read/write come from (some executors may have particular demands!), and which executor to submit requests to. Signed-off-by: Tom Burdick <thomas.burdick@intel.com>
2019-06-26 17:17:18 +02:00
#include <zephyr/rtio/rtio_executor_simple.h>
#include <zephyr/rtio/rtio.h>
#include <zephyr/kernel.h>
#include <zephyr/logging/log.h>
LOG_MODULE_REGISTER(rtio_executor_simple, CONFIG_RTIO_LOG_LEVEL);
/**
* @brief Submit submissions to simple executor
*
* The simple executor provides no concurrency instead
* execution each submission chain one after the next.
*
* @param r RTIO context
*
* @retval 0 Always succeeds
*/
int rtio_simple_submit(struct rtio *r)
{
rtio: Shareable lock-free iodevs By using an mpsc queue for each iodev, the iodev itself is shareable across contexts. Since its lock free, submits may occur even from an ISR context. Rather than a fixed size queue, and with it the possibility of running out of pre-allocated spots, each iodev now holds a wait-free mpsc queue head. This changes the parameter of iodev submit to be a struct containing 4 pointers for the rtio context, the submission queue entry, and the mpsc node for the iodevs submission queue. This solves the problem involving busy iodevs working with real devices. For example a busy SPI bus driver could enqueue, without locking, a request to start once the current request is done. The queue entries are expected to be owned and allocated by the executor rather than the iodev. This helps simplify potential tuning knobs to one place, the RTIO context and its executor an application directly uses. As the test case shows iodevs can operate effectively lock free with the mpsc queue and a single atomic denoting the current task. Signed-off-by: Tom Burdick <thomas.burdick@intel.com>
2022-10-28 04:44:00 +02:00
struct rtio_simple_executor *exc = (struct rtio_simple_executor *)r->executor;
/* Task is already running */
if (exc->task.sqe != NULL) {
return 0;
}
rtio: Real-Time Input/Output Stream A DMA friendly Stream API for zephyr. Based on ideas from io_uring and iio, a queue based API for I/O operations. Provides a pair of fixed length ringbuffer backed queues for submitting I/O requests and recieving I/O completions. The requests may be chained together to ensure the next operation does not start until the current one is complete. Requests target an abstract rtio_iodev which is expected to wrap all the hardware particulars of how to perform the operation. For example with a SPI bus device, a description of what a read, and write mean can be decided by the iodev wrapping a particular device hanging off of a SPI controller. The queue pair are submitted to an executor which may be a simple inplace looping executor done in the callers execution context (thread/stack) but other executors are expected. A threadpool executor might for example allow for concurrent request chains to execute in parallel. A DMA executor, in conjunction with DMA aware iodevs would allow for hardware offloading of operations going so far as to schedule with priority using hardware arbitration. Both the iodev and executor are definable by a particular SoC, meaning they can work in conjuction to perform IO operations using a particular DMA controller or methodology if desired. The application decides entirely how large the queues are, where the buffers to read/write come from (some executors may have particular demands!), and which executor to submit requests to. Signed-off-by: Tom Burdick <thomas.burdick@intel.com>
2019-06-26 17:17:18 +02:00
struct rtio_sqe *sqe = rtio_spsc_consume(r->sq);
rtio: Add transactional submissions Transactional submissions treat a sequence of sqes as a single atomic submission given to a single iodev and expect as a reply a single completion. This is useful for scatter gather like APIs that exist in Zephyr already for I2C and SPI. Signed-off-by: Tom Burdick <thomas.burdick@intel.com>
2023-02-25 00:43:59 +01:00
if (sqe == NULL) {
return 0;
}
/* Some validation on the sqe to ensure no programming errors were
* made so assumptions in ok and err are valid.
*/
#ifdef CONFIG_ASSERT
__ASSERT(sqe != NULL, "Expected a valid sqe on submit call");
bool transaction = sqe->flags & RTIO_SQE_TRANSACTION;
bool chained = sqe->flags & RTIO_SQE_CHAINED;
if (transaction || chained) {
struct rtio_sqe *next = rtio_spsc_next(r->sq, sqe);
__ASSERT(next != NULL,
"sqe %p flagged as transaction (%d) or chained (%d) without subsequent sqe in queue",
sqe, transaction, chained);
}
__ASSERT(!(chained && transaction),
"sqe %p flagged as both being transaction and chained, only one is allowed",
sqe);
#endif
rtio: Shareable lock-free iodevs By using an mpsc queue for each iodev, the iodev itself is shareable across contexts. Since its lock free, submits may occur even from an ISR context. Rather than a fixed size queue, and with it the possibility of running out of pre-allocated spots, each iodev now holds a wait-free mpsc queue head. This changes the parameter of iodev submit to be a struct containing 4 pointers for the rtio context, the submission queue entry, and the mpsc node for the iodevs submission queue. This solves the problem involving busy iodevs working with real devices. For example a busy SPI bus driver could enqueue, without locking, a request to start once the current request is done. The queue entries are expected to be owned and allocated by the executor rather than the iodev. This helps simplify potential tuning knobs to one place, the RTIO context and its executor an application directly uses. As the test case shows iodevs can operate effectively lock free with the mpsc queue and a single atomic denoting the current task. Signed-off-by: Tom Burdick <thomas.burdick@intel.com>
2022-10-28 04:44:00 +02:00
exc->task.sqe = sqe;
exc->task.r = r;
rtio: Add callback op Adds a callback op to RTIO enabling C logic to be interspersed with I/O operations. This is not safe from userspace but allowable in kernel space. Signed-off-by: Tom Burdick <thomas.burdick@intel.com>
2023-02-17 19:16:05 +01:00
rtio_executor_submit(&exc->task);
rtio: Real-Time Input/Output Stream A DMA friendly Stream API for zephyr. Based on ideas from io_uring and iio, a queue based API for I/O operations. Provides a pair of fixed length ringbuffer backed queues for submitting I/O requests and recieving I/O completions. The requests may be chained together to ensure the next operation does not start until the current one is complete. Requests target an abstract rtio_iodev which is expected to wrap all the hardware particulars of how to perform the operation. For example with a SPI bus device, a description of what a read, and write mean can be decided by the iodev wrapping a particular device hanging off of a SPI controller. The queue pair are submitted to an executor which may be a simple inplace looping executor done in the callers execution context (thread/stack) but other executors are expected. A threadpool executor might for example allow for concurrent request chains to execute in parallel. A DMA executor, in conjunction with DMA aware iodevs would allow for hardware offloading of operations going so far as to schedule with priority using hardware arbitration. Both the iodev and executor are definable by a particular SoC, meaning they can work in conjuction to perform IO operations using a particular DMA controller or methodology if desired. The application decides entirely how large the queues are, where the buffers to read/write come from (some executors may have particular demands!), and which executor to submit requests to. Signed-off-by: Tom Burdick <thomas.burdick@intel.com>
2019-06-26 17:17:18 +02:00
return 0;
}
/**
* @brief Callback from an iodev describing success
*/
rtio: Shareable lock-free iodevs By using an mpsc queue for each iodev, the iodev itself is shareable across contexts. Since its lock free, submits may occur even from an ISR context. Rather than a fixed size queue, and with it the possibility of running out of pre-allocated spots, each iodev now holds a wait-free mpsc queue head. This changes the parameter of iodev submit to be a struct containing 4 pointers for the rtio context, the submission queue entry, and the mpsc node for the iodevs submission queue. This solves the problem involving busy iodevs working with real devices. For example a busy SPI bus driver could enqueue, without locking, a request to start once the current request is done. The queue entries are expected to be owned and allocated by the executor rather than the iodev. This helps simplify potential tuning knobs to one place, the RTIO context and its executor an application directly uses. As the test case shows iodevs can operate effectively lock free with the mpsc queue and a single atomic denoting the current task. Signed-off-by: Tom Burdick <thomas.burdick@intel.com>
2022-10-28 04:44:00 +02:00
void rtio_simple_ok(struct rtio_iodev_sqe *iodev_sqe, int result)
rtio: Real-Time Input/Output Stream A DMA friendly Stream API for zephyr. Based on ideas from io_uring and iio, a queue based API for I/O operations. Provides a pair of fixed length ringbuffer backed queues for submitting I/O requests and recieving I/O completions. The requests may be chained together to ensure the next operation does not start until the current one is complete. Requests target an abstract rtio_iodev which is expected to wrap all the hardware particulars of how to perform the operation. For example with a SPI bus device, a description of what a read, and write mean can be decided by the iodev wrapping a particular device hanging off of a SPI controller. The queue pair are submitted to an executor which may be a simple inplace looping executor done in the callers execution context (thread/stack) but other executors are expected. A threadpool executor might for example allow for concurrent request chains to execute in parallel. A DMA executor, in conjunction with DMA aware iodevs would allow for hardware offloading of operations going so far as to schedule with priority using hardware arbitration. Both the iodev and executor are definable by a particular SoC, meaning they can work in conjuction to perform IO operations using a particular DMA controller or methodology if desired. The application decides entirely how large the queues are, where the buffers to read/write come from (some executors may have particular demands!), and which executor to submit requests to. Signed-off-by: Tom Burdick <thomas.burdick@intel.com>
2019-06-26 17:17:18 +02:00
{
rtio: Shareable lock-free iodevs By using an mpsc queue for each iodev, the iodev itself is shareable across contexts. Since its lock free, submits may occur even from an ISR context. Rather than a fixed size queue, and with it the possibility of running out of pre-allocated spots, each iodev now holds a wait-free mpsc queue head. This changes the parameter of iodev submit to be a struct containing 4 pointers for the rtio context, the submission queue entry, and the mpsc node for the iodevs submission queue. This solves the problem involving busy iodevs working with real devices. For example a busy SPI bus driver could enqueue, without locking, a request to start once the current request is done. The queue entries are expected to be owned and allocated by the executor rather than the iodev. This helps simplify potential tuning knobs to one place, the RTIO context and its executor an application directly uses. As the test case shows iodevs can operate effectively lock free with the mpsc queue and a single atomic denoting the current task. Signed-off-by: Tom Burdick <thomas.burdick@intel.com>
2022-10-28 04:44:00 +02:00
struct rtio *r = iodev_sqe->r;
const struct rtio_sqe *sqe = iodev_sqe->sqe;
#ifdef CONFIG_ASSERT
struct rtio_simple_executor *exc =
(struct rtio_simple_executor *)r->executor;
__ASSERT_NO_MSG(iodev_sqe == &exc->task);
#endif
rtio: Add transactional submissions Transactional submissions treat a sequence of sqes as a single atomic submission given to a single iodev and expect as a reply a single completion. This is useful for scatter gather like APIs that exist in Zephyr already for I2C and SPI. Signed-off-by: Tom Burdick <thomas.burdick@intel.com>
2023-02-25 00:43:59 +01:00
bool transaction = sqe->flags & RTIO_SQE_TRANSACTION;
while (transaction) {
rtio_spsc_release(r->sq);
sqe = rtio_spsc_consume(r->sq);
__ASSERT_NO_MSG(sqe != NULL);
transaction = sqe->flags & RTIO_SQE_TRANSACTION;
}
rtio: Release sqe before submitting cqe When an application is waiting on a completion it may be expecting, rightfully so, that a new submissions slot is available. Frees the submission queue event prior to enqueuing the completion queue event in the simple executor. Signed-off-by: Tom Burdick <thomas.burdick@intel.com>
2022-10-03 19:44:46 +02:00
void *userdata = sqe->userdata;
rtio: Real-Time Input/Output Stream A DMA friendly Stream API for zephyr. Based on ideas from io_uring and iio, a queue based API for I/O operations. Provides a pair of fixed length ringbuffer backed queues for submitting I/O requests and recieving I/O completions. The requests may be chained together to ensure the next operation does not start until the current one is complete. Requests target an abstract rtio_iodev which is expected to wrap all the hardware particulars of how to perform the operation. For example with a SPI bus device, a description of what a read, and write mean can be decided by the iodev wrapping a particular device hanging off of a SPI controller. The queue pair are submitted to an executor which may be a simple inplace looping executor done in the callers execution context (thread/stack) but other executors are expected. A threadpool executor might for example allow for concurrent request chains to execute in parallel. A DMA executor, in conjunction with DMA aware iodevs would allow for hardware offloading of operations going so far as to schedule with priority using hardware arbitration. Both the iodev and executor are definable by a particular SoC, meaning they can work in conjuction to perform IO operations using a particular DMA controller or methodology if desired. The application decides entirely how large the queues are, where the buffers to read/write come from (some executors may have particular demands!), and which executor to submit requests to. Signed-off-by: Tom Burdick <thomas.burdick@intel.com>
2019-06-26 17:17:18 +02:00
rtio_spsc_release(r->sq);
rtio: Shareable lock-free iodevs By using an mpsc queue for each iodev, the iodev itself is shareable across contexts. Since its lock free, submits may occur even from an ISR context. Rather than a fixed size queue, and with it the possibility of running out of pre-allocated spots, each iodev now holds a wait-free mpsc queue head. This changes the parameter of iodev submit to be a struct containing 4 pointers for the rtio context, the submission queue entry, and the mpsc node for the iodevs submission queue. This solves the problem involving busy iodevs working with real devices. For example a busy SPI bus driver could enqueue, without locking, a request to start once the current request is done. The queue entries are expected to be owned and allocated by the executor rather than the iodev. This helps simplify potential tuning knobs to one place, the RTIO context and its executor an application directly uses. As the test case shows iodevs can operate effectively lock free with the mpsc queue and a single atomic denoting the current task. Signed-off-by: Tom Burdick <thomas.burdick@intel.com>
2022-10-28 04:44:00 +02:00
iodev_sqe->sqe = NULL;
rtio: Add transactional submissions Transactional submissions treat a sequence of sqes as a single atomic submission given to a single iodev and expect as a reply a single completion. This is useful for scatter gather like APIs that exist in Zephyr already for I2C and SPI. Signed-off-by: Tom Burdick <thomas.burdick@intel.com>
2023-02-25 00:43:59 +01:00
rtio: Release sqe before submitting cqe When an application is waiting on a completion it may be expecting, rightfully so, that a new submissions slot is available. Frees the submission queue event prior to enqueuing the completion queue event in the simple executor. Signed-off-by: Tom Burdick <thomas.burdick@intel.com>
2022-10-03 19:44:46 +02:00
rtio_cqe_submit(r, result, userdata);
rtio: Real-Time Input/Output Stream A DMA friendly Stream API for zephyr. Based on ideas from io_uring and iio, a queue based API for I/O operations. Provides a pair of fixed length ringbuffer backed queues for submitting I/O requests and recieving I/O completions. The requests may be chained together to ensure the next operation does not start until the current one is complete. Requests target an abstract rtio_iodev which is expected to wrap all the hardware particulars of how to perform the operation. For example with a SPI bus device, a description of what a read, and write mean can be decided by the iodev wrapping a particular device hanging off of a SPI controller. The queue pair are submitted to an executor which may be a simple inplace looping executor done in the callers execution context (thread/stack) but other executors are expected. A threadpool executor might for example allow for concurrent request chains to execute in parallel. A DMA executor, in conjunction with DMA aware iodevs would allow for hardware offloading of operations going so far as to schedule with priority using hardware arbitration. Both the iodev and executor are definable by a particular SoC, meaning they can work in conjuction to perform IO operations using a particular DMA controller or methodology if desired. The application decides entirely how large the queues are, where the buffers to read/write come from (some executors may have particular demands!), and which executor to submit requests to. Signed-off-by: Tom Burdick <thomas.burdick@intel.com>
2019-06-26 17:17:18 +02:00
rtio_simple_submit(r);
}
/**
* @brief Callback from an iodev describing error
rtio: Add transactional submissions Transactional submissions treat a sequence of sqes as a single atomic submission given to a single iodev and expect as a reply a single completion. This is useful for scatter gather like APIs that exist in Zephyr already for I2C and SPI. Signed-off-by: Tom Burdick <thomas.burdick@intel.com>
2023-02-25 00:43:59 +01:00
*
* Some assumptions are made and should have been validated on rtio_submit
* - a sqe marked as chained or transaction has a next sqe
* - a sqe is marked either chained or transaction but not both
rtio: Real-Time Input/Output Stream A DMA friendly Stream API for zephyr. Based on ideas from io_uring and iio, a queue based API for I/O operations. Provides a pair of fixed length ringbuffer backed queues for submitting I/O requests and recieving I/O completions. The requests may be chained together to ensure the next operation does not start until the current one is complete. Requests target an abstract rtio_iodev which is expected to wrap all the hardware particulars of how to perform the operation. For example with a SPI bus device, a description of what a read, and write mean can be decided by the iodev wrapping a particular device hanging off of a SPI controller. The queue pair are submitted to an executor which may be a simple inplace looping executor done in the callers execution context (thread/stack) but other executors are expected. A threadpool executor might for example allow for concurrent request chains to execute in parallel. A DMA executor, in conjunction with DMA aware iodevs would allow for hardware offloading of operations going so far as to schedule with priority using hardware arbitration. Both the iodev and executor are definable by a particular SoC, meaning they can work in conjuction to perform IO operations using a particular DMA controller or methodology if desired. The application decides entirely how large the queues are, where the buffers to read/write come from (some executors may have particular demands!), and which executor to submit requests to. Signed-off-by: Tom Burdick <thomas.burdick@intel.com>
2019-06-26 17:17:18 +02:00
*/
rtio: Shareable lock-free iodevs By using an mpsc queue for each iodev, the iodev itself is shareable across contexts. Since its lock free, submits may occur even from an ISR context. Rather than a fixed size queue, and with it the possibility of running out of pre-allocated spots, each iodev now holds a wait-free mpsc queue head. This changes the parameter of iodev submit to be a struct containing 4 pointers for the rtio context, the submission queue entry, and the mpsc node for the iodevs submission queue. This solves the problem involving busy iodevs working with real devices. For example a busy SPI bus driver could enqueue, without locking, a request to start once the current request is done. The queue entries are expected to be owned and allocated by the executor rather than the iodev. This helps simplify potential tuning knobs to one place, the RTIO context and its executor an application directly uses. As the test case shows iodevs can operate effectively lock free with the mpsc queue and a single atomic denoting the current task. Signed-off-by: Tom Burdick <thomas.burdick@intel.com>
2022-10-28 04:44:00 +02:00
void rtio_simple_err(struct rtio_iodev_sqe *iodev_sqe, int result)
rtio: Real-Time Input/Output Stream A DMA friendly Stream API for zephyr. Based on ideas from io_uring and iio, a queue based API for I/O operations. Provides a pair of fixed length ringbuffer backed queues for submitting I/O requests and recieving I/O completions. The requests may be chained together to ensure the next operation does not start until the current one is complete. Requests target an abstract rtio_iodev which is expected to wrap all the hardware particulars of how to perform the operation. For example with a SPI bus device, a description of what a read, and write mean can be decided by the iodev wrapping a particular device hanging off of a SPI controller. The queue pair are submitted to an executor which may be a simple inplace looping executor done in the callers execution context (thread/stack) but other executors are expected. A threadpool executor might for example allow for concurrent request chains to execute in parallel. A DMA executor, in conjunction with DMA aware iodevs would allow for hardware offloading of operations going so far as to schedule with priority using hardware arbitration. Both the iodev and executor are definable by a particular SoC, meaning they can work in conjuction to perform IO operations using a particular DMA controller or methodology if desired. The application decides entirely how large the queues are, where the buffers to read/write come from (some executors may have particular demands!), and which executor to submit requests to. Signed-off-by: Tom Burdick <thomas.burdick@intel.com>
2019-06-26 17:17:18 +02:00
{
rtio: Add transactional submissions Transactional submissions treat a sequence of sqes as a single atomic submission given to a single iodev and expect as a reply a single completion. This is useful for scatter gather like APIs that exist in Zephyr already for I2C and SPI. Signed-off-by: Tom Burdick <thomas.burdick@intel.com>
2023-02-25 00:43:59 +01:00
const struct rtio_sqe *sqe = iodev_sqe->sqe;
rtio: Shareable lock-free iodevs By using an mpsc queue for each iodev, the iodev itself is shareable across contexts. Since its lock free, submits may occur even from an ISR context. Rather than a fixed size queue, and with it the possibility of running out of pre-allocated spots, each iodev now holds a wait-free mpsc queue head. This changes the parameter of iodev submit to be a struct containing 4 pointers for the rtio context, the submission queue entry, and the mpsc node for the iodevs submission queue. This solves the problem involving busy iodevs working with real devices. For example a busy SPI bus driver could enqueue, without locking, a request to start once the current request is done. The queue entries are expected to be owned and allocated by the executor rather than the iodev. This helps simplify potential tuning knobs to one place, the RTIO context and its executor an application directly uses. As the test case shows iodevs can operate effectively lock free with the mpsc queue and a single atomic denoting the current task. Signed-off-by: Tom Burdick <thomas.burdick@intel.com>
2022-10-28 04:44:00 +02:00
struct rtio *r = iodev_sqe->r;
rtio: Release sqe before submitting cqe When an application is waiting on a completion it may be expecting, rightfully so, that a new submissions slot is available. Frees the submission queue event prior to enqueuing the completion queue event in the simple executor. Signed-off-by: Tom Burdick <thomas.burdick@intel.com>
2022-10-03 19:44:46 +02:00
void *userdata = sqe->userdata;
bool chained = sqe->flags & RTIO_SQE_CHAINED;
rtio: Add transactional submissions Transactional submissions treat a sequence of sqes as a single atomic submission given to a single iodev and expect as a reply a single completion. This is useful for scatter gather like APIs that exist in Zephyr already for I2C and SPI. Signed-off-by: Tom Burdick <thomas.burdick@intel.com>
2023-02-25 00:43:59 +01:00
bool transaction = sqe->flags & RTIO_SQE_TRANSACTION;
rtio: Real-Time Input/Output Stream A DMA friendly Stream API for zephyr. Based on ideas from io_uring and iio, a queue based API for I/O operations. Provides a pair of fixed length ringbuffer backed queues for submitting I/O requests and recieving I/O completions. The requests may be chained together to ensure the next operation does not start until the current one is complete. Requests target an abstract rtio_iodev which is expected to wrap all the hardware particulars of how to perform the operation. For example with a SPI bus device, a description of what a read, and write mean can be decided by the iodev wrapping a particular device hanging off of a SPI controller. The queue pair are submitted to an executor which may be a simple inplace looping executor done in the callers execution context (thread/stack) but other executors are expected. A threadpool executor might for example allow for concurrent request chains to execute in parallel. A DMA executor, in conjunction with DMA aware iodevs would allow for hardware offloading of operations going so far as to schedule with priority using hardware arbitration. Both the iodev and executor are definable by a particular SoC, meaning they can work in conjuction to perform IO operations using a particular DMA controller or methodology if desired. The application decides entirely how large the queues are, where the buffers to read/write come from (some executors may have particular demands!), and which executor to submit requests to. Signed-off-by: Tom Burdick <thomas.burdick@intel.com>
2019-06-26 17:17:18 +02:00
rtio: Shareable lock-free iodevs By using an mpsc queue for each iodev, the iodev itself is shareable across contexts. Since its lock free, submits may occur even from an ISR context. Rather than a fixed size queue, and with it the possibility of running out of pre-allocated spots, each iodev now holds a wait-free mpsc queue head. This changes the parameter of iodev submit to be a struct containing 4 pointers for the rtio context, the submission queue entry, and the mpsc node for the iodevs submission queue. This solves the problem involving busy iodevs working with real devices. For example a busy SPI bus driver could enqueue, without locking, a request to start once the current request is done. The queue entries are expected to be owned and allocated by the executor rather than the iodev. This helps simplify potential tuning knobs to one place, the RTIO context and its executor an application directly uses. As the test case shows iodevs can operate effectively lock free with the mpsc queue and a single atomic denoting the current task. Signed-off-by: Tom Burdick <thomas.burdick@intel.com>
2022-10-28 04:44:00 +02:00
#ifdef CONFIG_ASSERT
struct rtio_simple_executor *exc =
(struct rtio_simple_executor *)r->executor;
__ASSERT_NO_MSG(iodev_sqe == &exc->task);
#endif
rtio: Real-Time Input/Output Stream A DMA friendly Stream API for zephyr. Based on ideas from io_uring and iio, a queue based API for I/O operations. Provides a pair of fixed length ringbuffer backed queues for submitting I/O requests and recieving I/O completions. The requests may be chained together to ensure the next operation does not start until the current one is complete. Requests target an abstract rtio_iodev which is expected to wrap all the hardware particulars of how to perform the operation. For example with a SPI bus device, a description of what a read, and write mean can be decided by the iodev wrapping a particular device hanging off of a SPI controller. The queue pair are submitted to an executor which may be a simple inplace looping executor done in the callers execution context (thread/stack) but other executors are expected. A threadpool executor might for example allow for concurrent request chains to execute in parallel. A DMA executor, in conjunction with DMA aware iodevs would allow for hardware offloading of operations going so far as to schedule with priority using hardware arbitration. Both the iodev and executor are definable by a particular SoC, meaning they can work in conjuction to perform IO operations using a particular DMA controller or methodology if desired. The application decides entirely how large the queues are, where the buffers to read/write come from (some executors may have particular demands!), and which executor to submit requests to. Signed-off-by: Tom Burdick <thomas.burdick@intel.com>
2019-06-26 17:17:18 +02:00
rtio_spsc_release(r->sq);
rtio: Shareable lock-free iodevs By using an mpsc queue for each iodev, the iodev itself is shareable across contexts. Since its lock free, submits may occur even from an ISR context. Rather than a fixed size queue, and with it the possibility of running out of pre-allocated spots, each iodev now holds a wait-free mpsc queue head. This changes the parameter of iodev submit to be a struct containing 4 pointers for the rtio context, the submission queue entry, and the mpsc node for the iodevs submission queue. This solves the problem involving busy iodevs working with real devices. For example a busy SPI bus driver could enqueue, without locking, a request to start once the current request is done. The queue entries are expected to be owned and allocated by the executor rather than the iodev. This helps simplify potential tuning knobs to one place, the RTIO context and its executor an application directly uses. As the test case shows iodevs can operate effectively lock free with the mpsc queue and a single atomic denoting the current task. Signed-off-by: Tom Burdick <thomas.burdick@intel.com>
2022-10-28 04:44:00 +02:00
iodev_sqe->sqe = NULL;
rtio: Add transactional submissions Transactional submissions treat a sequence of sqes as a single atomic submission given to a single iodev and expect as a reply a single completion. This is useful for scatter gather like APIs that exist in Zephyr already for I2C and SPI. Signed-off-by: Tom Burdick <thomas.burdick@intel.com>
2023-02-25 00:43:59 +01:00
if (!transaction) {
rtio_cqe_submit(r, result, userdata);
}
while (chained | transaction) {
sqe = rtio_spsc_consume(r->sq);
chained = sqe->flags & RTIO_SQE_CHAINED;
transaction = sqe->flags & RTIO_SQE_TRANSACTION;
userdata = sqe->userdata;
rtio_spsc_release(r->sq);
if (!transaction) {
rtio_cqe_submit(r, result, userdata);
} else {
rtio: Release sqe before submitting cqe When an application is waiting on a completion it may be expecting, rightfully so, that a new submissions slot is available. Frees the submission queue event prior to enqueuing the completion queue event in the simple executor. Signed-off-by: Tom Burdick <thomas.burdick@intel.com>
2022-10-03 19:44:46 +02:00
rtio_cqe_submit(r, -ECANCELED, userdata);
rtio: Real-Time Input/Output Stream A DMA friendly Stream API for zephyr. Based on ideas from io_uring and iio, a queue based API for I/O operations. Provides a pair of fixed length ringbuffer backed queues for submitting I/O requests and recieving I/O completions. The requests may be chained together to ensure the next operation does not start until the current one is complete. Requests target an abstract rtio_iodev which is expected to wrap all the hardware particulars of how to perform the operation. For example with a SPI bus device, a description of what a read, and write mean can be decided by the iodev wrapping a particular device hanging off of a SPI controller. The queue pair are submitted to an executor which may be a simple inplace looping executor done in the callers execution context (thread/stack) but other executors are expected. A threadpool executor might for example allow for concurrent request chains to execute in parallel. A DMA executor, in conjunction with DMA aware iodevs would allow for hardware offloading of operations going so far as to schedule with priority using hardware arbitration. Both the iodev and executor are definable by a particular SoC, meaning they can work in conjuction to perform IO operations using a particular DMA controller or methodology if desired. The application decides entirely how large the queues are, where the buffers to read/write come from (some executors may have particular demands!), and which executor to submit requests to. Signed-off-by: Tom Burdick <thomas.burdick@intel.com>
2019-06-26 17:17:18 +02:00
}
rtio: Add transactional submissions Transactional submissions treat a sequence of sqes as a single atomic submission given to a single iodev and expect as a reply a single completion. This is useful for scatter gather like APIs that exist in Zephyr already for I2C and SPI. Signed-off-by: Tom Burdick <thomas.burdick@intel.com>
2023-02-25 00:43:59 +01:00
}
rtio: Real-Time Input/Output Stream A DMA friendly Stream API for zephyr. Based on ideas from io_uring and iio, a queue based API for I/O operations. Provides a pair of fixed length ringbuffer backed queues for submitting I/O requests and recieving I/O completions. The requests may be chained together to ensure the next operation does not start until the current one is complete. Requests target an abstract rtio_iodev which is expected to wrap all the hardware particulars of how to perform the operation. For example with a SPI bus device, a description of what a read, and write mean can be decided by the iodev wrapping a particular device hanging off of a SPI controller. The queue pair are submitted to an executor which may be a simple inplace looping executor done in the callers execution context (thread/stack) but other executors are expected. A threadpool executor might for example allow for concurrent request chains to execute in parallel. A DMA executor, in conjunction with DMA aware iodevs would allow for hardware offloading of operations going so far as to schedule with priority using hardware arbitration. Both the iodev and executor are definable by a particular SoC, meaning they can work in conjuction to perform IO operations using a particular DMA controller or methodology if desired. The application decides entirely how large the queues are, where the buffers to read/write come from (some executors may have particular demands!), and which executor to submit requests to. Signed-off-by: Tom Burdick <thomas.burdick@intel.com>
2019-06-26 17:17:18 +02:00
rtio: Add transactional submissions Transactional submissions treat a sequence of sqes as a single atomic submission given to a single iodev and expect as a reply a single completion. This is useful for scatter gather like APIs that exist in Zephyr already for I2C and SPI. Signed-off-by: Tom Burdick <thomas.burdick@intel.com>
2023-02-25 00:43:59 +01:00
iodev_sqe->sqe = rtio_spsc_consume(r->sq);
if (iodev_sqe->sqe != NULL) {
rtio: Add callback op Adds a callback op to RTIO enabling C logic to be interspersed with I/O operations. This is not safe from userspace but allowable in kernel space. Signed-off-by: Tom Burdick <thomas.burdick@intel.com>
2023-02-17 19:16:05 +01:00
rtio_executor_submit(iodev_sqe);
rtio: Real-Time Input/Output Stream A DMA friendly Stream API for zephyr. Based on ideas from io_uring and iio, a queue based API for I/O operations. Provides a pair of fixed length ringbuffer backed queues for submitting I/O requests and recieving I/O completions. The requests may be chained together to ensure the next operation does not start until the current one is complete. Requests target an abstract rtio_iodev which is expected to wrap all the hardware particulars of how to perform the operation. For example with a SPI bus device, a description of what a read, and write mean can be decided by the iodev wrapping a particular device hanging off of a SPI controller. The queue pair are submitted to an executor which may be a simple inplace looping executor done in the callers execution context (thread/stack) but other executors are expected. A threadpool executor might for example allow for concurrent request chains to execute in parallel. A DMA executor, in conjunction with DMA aware iodevs would allow for hardware offloading of operations going so far as to schedule with priority using hardware arbitration. Both the iodev and executor are definable by a particular SoC, meaning they can work in conjuction to perform IO operations using a particular DMA controller or methodology if desired. The application decides entirely how large the queues are, where the buffers to read/write come from (some executors may have particular demands!), and which executor to submit requests to. Signed-off-by: Tom Burdick <thomas.burdick@intel.com>
2019-06-26 17:17:18 +02:00
}
}
Reference in a new issue Copy permalink