Re: [ECOS] Avoiding memcpy in ethernet drivers

[Bart Veer <bartv at ecoscentric dot com>]

>>>>> "Edgar" == Edgar Grimberg <edgar.grimberg@gmail.com> writes:

    Edgar> Having to write a driver for an ethernet device, I noticed
    Edgar> that there are 2 memcpy()s in most of the drivers, one when
    Edgar> sending and one when receiving. This usually happens from
    Edgar> and to the sg_list passed from the hardware independent
    Edgar> ethernet driver. Now, is this really necessary?

Ethernet hardware varies widely, but there are two main categories:
fifo vs. DMA.

With a fifo-based device the driver just puts an outgoing packet into
a transmit fifo, possibly an int at a time, possibly a byte at a time.
Similarly incoming packets are read out of a receive fifo. So there
has to be a copy operation between the sg_list and the fifo.

With DMA, ideally there would be no need for a copy operation - if the
hardware was designed just right, and if higher-level code like the
BSD, RedBoot or LWIP TCP/IP stacks did the right thing. Unfortunately
neither assumption is safe.

For outgoing packets the DMA engine typically imposes some
limitations. For example the start of a DMA buffer may have to be
aligned to an integer boundary, or to a cacheline boundary (typically
but not always 16-bytes). If an outgoing packet is split over multiple
buffers then the DMA engine may require that every buffer except the
last one is a multiple of the cacheline size. Currently there is no
mechanism for ethernet drivers to indicate such restrictions to
higher-level code, and the sg_list's provided by higher-level code may
provide buffers with arbitrary alignments and lengths.

So, typically ethernet drivers will copy the sg_list into a single
contiguous buffer aligned to a suitable boundary. Sometimes that copy
won't be necessary, but if you are not careful then you'll spend more
time figuring out which bits need copying and which don't than it
would take to just do the copy anyway. If you want to eliminate the
copy at the device driver level then you need to provide a way for
device drivers to indicate the h/w constraints to higher-level code,
and you need to update the higher-level code so that it will guarantee
that the constraints are always satisfied.

For incoming packets, the current API between device drivers and
higher-level code assumes that the driver knows the size of the
packet. With DMA engines that info is not available in time. Instead
typically the driver has a static full-sized receive buffer, and once
a full packet has been received it will allocate an sg_list and copy
from the static buffer. Using a static buffer also avoids problems
with DMA engine restrictions, e.g. it is possible to statically
allocate a buffer that is cacheline-aligned and a multiple of the
cacheline size.

Changing the current API between drivers and higher-level code would
impact existing drivers. There are a number of different ways of
tackling that problem, but nothing simple.


The current code works well enough for most people. Realistically, it
is unlikely that anything would happen in this area except possibly as
part of a re-import of the BSD stack. Just the re-import would be a
big job, never mind changing the device driver interface. It may
happen some day, most likely if/when somebody really needs maximum
performance and is willing to pay for the work.

Bart

-- 
Bart Veer                                   eCos Configuration Architect
eCosCentric Limited    The eCos experts      http://www.ecoscentric.com/
Barnwell House, Barnwell Drive, Cambridge, UK.      Tel: +44 1223 245571
Registered in England and Wales: Reg No 4422071.
Besuchen Sie uns vom 3.-5.03.09 auf der Embedded World 2009, Stand 11-300
Visit us at Embedded World 2009, Nürnberg, Germany, 3-5 Mar, Stand 11-300

-- 
Before posting, please read the FAQ: http://ecos.sourceware.org/fom/ecos
and search the list archive: http://ecos.sourceware.org/ml/ecos-discuss