2. Rights to Copy
License: Creative Commons Attribution - Share Alike 3.0
You are free:
to copy, distribute, display, and perform the work
to make derivative works
to make commercial use of the work
Under the following conditions:
Attribution – Derived from original work of Free Electrons
Share Alike
For any reuse or distribution, you must make clear to others the license terms
of this work.
September 7, 2017
Embedded Linux Network Device Driver
Development
2
4. Objectives
Understanding the structure of ethernet and WIFI device drivers
Developing Linux network device drivers
Embedded Linux Network Device Driver
Development
4 September 7, 2017
5. Prerequisites
Solid C programming
Knowledge of Linux commands is a plus
Knowledge of networking
Embedded Linux Kernel and Driver Development training or equivalent
Embedded Linux Network Device Driver
Development
5 September 7, 2017
6. Notes
Ask any time.
Turn your cell silent.
Embedded Linux Network Device Driver
Development
6 September 7, 2017
7. Outline
Introduction
Socket Buffers
Network Devices
Communication with Network Protocol
Communication with PHY
Buffer Management and Concurrency Issues
Network Throughput
Other Network Driver Considerations
Ethernet Driver vs WIFI Driver
September 7, 2017
Embedded Linux Network Device Driver
Development
7
8. Outline
Introduction
Socket Buffers
Network Devices
Communication with Network Protocol
Communication with PHY
Buffer Management and Concurrency Issues
Network Throughput
Other Network Driver Considerations
Ethernet Driver vs WIFI Driver
September 7, 2017
Embedded Linux Network Device Driver
Development
8
9. OSI Model and Linux Kernel
September 7, 2017
Embedded Linux Network Device Driver
Development
9
User Space
Kernel Network
Stack
Device Driver
11. Outline
Introduction
Socket Buffers
Network Devices
Communication with Network Protocol
Communication with PHY
Buffer Management and Concurrency Issues
Network Throughput
Other Network Driver Considerations
Ethernet Driver vs WIFI Driver
September 7, 2017
Embedded Linux Network Device Driver
Development
11
12. Socket Buffer (sk_buff)
struct sk_buff (include/linux/sk_buff.h) represents a network packet
Support data encapsulation/decapsulation through protocol layers
In addition to data, sk_buff maintains:
head, the start of the packet
data, the start of the packet payload
tail, the end of the packet payload
end, the end of the packet
len, the amount of data of the packet
These fields are updated when the packet goes through the protocol
layers
September 7, 2017
Embedded Linux Network Device Driver
Development
12
13. SKB Operations
Allocation
By dev_alloc_skb() function
Can be done in ISR
On Ethernet, allocated size = packet
length + 2
To word align IP header
IP header = 14 bytes
Reservation
By skb_reserve() function
Skipping NET_IP_ALIGN padding
bytes
September 7, 2017
Embedded Linux Network Device Driver
Development
13
14. SKB Operations cont’d
Data Copying
From DMA buffer to SKB during
reception for example
OR SKB to DMA buffer during
transmission (may be)
Pointers Update
After copying payload
September 7, 2017
Embedded Linux Network Device Driver
Development
14
15. Outline
Introduction
Socket Buffers
Network Devices
Communication with Network Protocol
Communication with PHY
Buffer Management and Concurrency Issues
Network Throughput
Other Network Driver Considerations
Ethernet Driver vs WIFI Driver
September 7, 2017
Embedded Linux Network Device Driver
Development
15
16. Network Device
struct net_device (include/linux/netdevice.h) represents a network
interface
Allocation takes place with alloc_netdev() function
Size of private data is passed as argument
Pointer to these private data can be read in net_device->priv
Use alloc_etherdev() for ethernet interfaces
Use alloc_ieee80211() for WIFI interfaces
September 7, 2017
Embedded Linux Network Device Driver
Development
16
17. Network Device cont’d
Registration with register_netdev() function
Unregistration with unregister_netdev() function
Liberation with free_netdev() function
September 7, 2017
Embedded Linux Network Device Driver
Development
17
18. Network Device Operations
Defined by struct net_device_ops (include/linux/netdevice.h)
Set the netdev_ops field in the struct net_device structure to point to the
struct net_device_ops structure
September 7, 2017
Embedded Linux Network Device Driver
Development
18
Operation Description
ndo_open() Called when network interface uped
ndo_close() Called when network interface downed
ndo_start_xmit() Start packet transmission
ndo_get_stats() Get statistics
ndo_do_ioctl() Implement device specific operations
ndo_set_rx_mode() Select promiscuous, multicast, etc
ndo_set_mac_address() Set MAC address
ndo_set_multicast_list() Set multicast filters
19. Network Device Operations cont’d
Sample operations are:
September 7, 2017
Embedded Linux Network Device Driver
Development
19
Operation Description
ndo_open() Called when network interface uped
ndo_close() Called when network interface downed
ndo_start_xmit() Start packet transmission
ndo_get_stats() Get statistics
ndo_do_ioctl() Implement device specific operations
ndo_set_rx_mode() Select promiscuous, multicast, etc
ndo_set_mac_address() Set MAC address
ndo_set_multicast_list() Set multicast filters
ndo_tx_timeout() Reset unresponsive network interface
20. Outline
Introduction
Socket Buffers
Network Devices
Communication with Network Protocol
Communication with PHY
Buffer Management and Concurrency Issues
Network Throughput
Other Network Driver Considerations
Ethernet Driver vs WIFI Driver
September 7, 2017
Embedded Linux Network Device Driver
Development
20
21. Flow Control Utility Functions
include/linux/netdevice.h
September 7, 2017
Embedded Linux Network Device Driver
Development
21
Operation Description
netif_start_queue() Tells kernel that driver is ready to send packets
netif_stop_queue() Tells kernel to stop sending packets (@cleanup or congestion)
netif_queue_stopped() Tells whether queue is stopped or not
netif_wake_queue() Wakes up a queue after a netif_stop_queue()
netif_rx() Gives SKB to kernel
22. Transmission
Kernel calls ndo_start_xmit() with SKB as argument
ndo_start_xmit():
1. Sets up DMA buffers and other HW mechanisms
2. Starts transmission
3. Can stop queueing if no more free DMA buffers available using
netif_stop_queue()
4. Returns NETDEV_TX_OK or NETDEV_TX_BUSY
When N packets have been sent, an interrupt is raised and the driver
should:
1. Acknowledging interrupt
2. Freeing used DMA buffers
3. Freeing SKB with dev_kfree_skb_irq()
4. If the queue was stopped, start it again
September 7, 2017
Embedded Linux Network Device Driver
Development
22
23. Reception: Non-NAPI
Reception is via an interrupt that should:
1. Allocate an SKB
2. Reserve NET_IP_ALIGN padding bytes
3. Copy packet from DMA buffers to SKB
4. Update SKB pointers
5. Update the skb->protocol field with eth_type_trans(skb, netdevice)
6. Give SKB to kernel by netif_rx()
netif_rx() use NET_RX_SOFTIRQ to offload the work of posting received data
packets to protocol layers
Nice and simple
@high traffic, interrupt rate is high
Solution: switch to polled mode when interrupt rate is too high (NAPI)
September 7, 2017
Embedded Linux Network Device Driver
Development
23
24. Reception: NAPI
In network interface private structure, add a struct napi_struct
(include/linux/netdevice.h)
@ driver initialization, register the NAPI poll operation
dev: network interface
&lp->napi: struct napi_struct
r6040_poll: NAPI poll operation
64: weight that represents the importance of network interface
September 7, 2017
Embedded Linux Network Device Driver
Development
24
25. Reception: NAPI cont’d
Rx ISR
Disables Rx interrupt and switch to
polled mode
if (napi_schedule_prep(&lp->napi)) {
/* Disable reception interrupts */
__napi_schedule(&lp->napi);
}
Kernel calls our poll() operation
regularly
Poll Function
static int r6040_poll(struct
napi_struct *napi, int budget)
Receives at most budget packets and
pushes them to the network stack
using netif_receive_skb()
If less than budget packets have been
received, switch back to interrupt
mode using napi_complete(&lp-
>napi) and reenable interrupts
Must return the number of packets
received
September 7, 2017
Embedded Linux Network Device Driver
Development
25
26. Outline
Introduction
Socket Buffers
Network Devices
Communication with Network Protocol
Communication with PHY
Buffer Management and Concurrency Issues
Network Throughput
Other Network Driver Considerations
Ethernet Driver vs WIFI Driver
September 7, 2017
Embedded Linux Network Device Driver
Development
26
27. Connection with PHY
MAC and PHY are connected using a MII or RMII interface
This interface contains two wires used for the MDIO bus
Driver needs to communicate with the PHY to get link information (up,
down, speed, full or half duplex) and configure the MAC accordingly
September 7, 2017
Embedded Linux Network Device Driver
Development
27
28. PHY in Kernel
Kernel has a framework (drivers/net/phy/) that
Exposes an API to communicate with PHY
Allows to implement PHY drivers
Implements a basic generic PHY driver that works with all PHY
See Documentation/networking/phy.txt
September 7, 2017
Embedded Linux Network Device Driver
Development
28
29. MDIO Bus Initialization
Driver creates a MDIO bus struct mii_bus (include/linux/mii.h) to tells PHY
infrastructure how to communicate with the PHY
mdio_read() and mdio_write are HW specific and must be implemented
by the driver
September 7, 2017
Embedded Linux Network Device Driver
Development
29
30. MDIO Bus Initialization cont’d
The ->irq[] array must be allocated and initialized
To use polling, set the values to PHY_POLL
Lp->mii_if->irq = kmalloc(sizeof(int)*PHY_MAX_ADDR, GFP_KERNEL);
for (i = 0; i < PHY_MAX_ADDR; i++)
lp->mii_if>irq[i] = PHY_POLL;
Finally, register the MDIO bus
This will scan the bus for PHYs and fill the mii_if->phy_map[] array with the
result
mdiobus_register(bp->mii_if);
September 7, 2017
Embedded Linux Network Device Driver
Development
30
31. Connecting with PHY
mdiobus_register() function filled the mii_if->phy_map[] array with struct
phy_device * pointers
Appropriate PHY (usually, only one is detected) must be selected
Connecting to the PHY allows to register a callback that will be called
when the link changes :
interface is usually PHY_INTERFACE_MODE_MII or
PHY_INTERFACE_MODE_RMII
September 7, 2017
Embedded Linux Network Device Driver
Development
31
32. Updating MAC Capabilities
MAC and PHY might have different capabilities
PHY handling Gigabit speed, but not MAC
Driver is responsible for updating phydev->advertise and phydev-
>supported to remove any PHY capability that the MAC doesn't support
September 7, 2017
Embedded Linux Network Device Driver
Development
32
33. Handling Link Changes
The callback that handle link changes should have the following prototype:
void foo_handle_link_change(struct net_device *dev)
It must check the duplex, speed and link fields of the struct phy_device
structure, and update Ethernet controller configuration accordingly
duplex is either DUPLEX_HALF or DUPLEX_FULL
speed is either SPEED_10, SPEED_100, SPEED_1000, SPEED_2500 or
SPEED_10000
link is a Boolean
September 7, 2017
Embedded Linux Network Device Driver
Development
33
34. Starting and stopping the PHY
After set up, to make the PHY driver poll regularly the PHY hardware, one
must start it with phy_start() functions
And stop it using phy_stop(), when no longer needed
September 7, 2017
Embedded Linux Network Device Driver
Development
34
35. Outline
Introduction
Socket Buffers
Network Devices
Communication with Network Protocol
Communication with PHY
Buffer Management and Concurrency Issues
Network Throughput
Other Network Driver Considerations
Ethernet Driver vs WIFI Driver
September 7, 2017
Embedded Linux Network Device Driver
Development
35
36. Buffers Management
During open(), driver pre-allocates needed DMA descriptos needed for
transmission and reception
When totally utilized, the driver tells the kernel to stop sending packets to
the driver
SKB are allocated and freed as per need
September 7, 2017
Embedded Linux Network Device Driver
Development
36
37. Concurrency
Access protection needed in the face of multiple execution threads:
Transmit thread
Receive thread
Transmit-complete interrupt
Receive interrupt
NAPI polling
September 7, 2017
Embedded Linux Network Device Driver
Development
37
38. Outline
Introduction
Socket Buffers
Network Devices
Communication with Network Protocol
Communication with PHY
Buffer Management and Concurrency Issues
Network Throughput
Other Network Driver Considerations
Ethernet Driver vs WIFI Driver
September 7, 2017
Embedded Linux Network Device Driver
Development
38
39. Network Benchmarking
Netperf (free from www.netperf.org ) can set up complex TCP/UDP
connection scenarios to control C/C’s such as protocol parameters,
number of simultaneous sessions, and size of data blocks.
Benchmarking = Comparing resulting throughput with maximum practical
bandwidth
Factors affecting throughput:
Driver implementation
Protocol used
September 7, 2017
Embedded Linux Network Device Driver
Development
39
40. Driver Performance
Execution time
Considering usage of DMA for large packets
Offloading CPU by moving functionalities to hardware (TCP checksum)
Taking into consideration upper layers behavior
September 7, 2017
Embedded Linux Network Device Driver
Development
40
41. Protocol Performance
TCP window size = data that can be transmitted before receiving ACK
Fast network interfaces + small window size = TCP sitting idle waiting for ACK
In UDP, the window size is not relevant as there is no ACK
Packet loss in TCP and UDP
Mapping protocol block sizes to MTU sizes
September 7, 2017
Embedded Linux Network Device Driver
Development
41
42. Outline
Introduction
Socket Buffers
Network Devices
Communication with Network Protocol
Communication with PHY
Buffer Management and Concurrency Issues
Network Throughput
Other Network Driver Considerations
Ethernet Driver vs WIFI Driver
September 7, 2017
Embedded Linux Network Device Driver
Development
42
43. Ethtool (include/linux/ethtool.h)
ethtool is a user space tool that query low-level information from an
Ethernet interface and to modify its configuration
@ driver, a struct ethtool_ops structure can be declared and connected to
the struct net_device using the ethtool_ops field
List of operations: get_settings(), set_settings(), get_drvinfo(), get_wol(),
set_wol(), get_link(), get_eeprom(), set_eeprom(), get_tso(), set_tso(),
get_flags(), set_flags(), etc.
Some of these operations can be implemented using the PHY interface
(phy_ethtool_gset(), phy_ethtool_sset()) or using generic
operations(ethtool_op_get_link() for example)
September 7, 2017
Embedded Linux Network Device Driver
Development
43
44. Statistics
The driver is responsible for keeping statistics up to date about the
number of packets/bytes received/transmitted, the number of errors, of
collisions, etc.
To expose these information, the driver must implement a get_stats()
operation, with the following prototype struct net_device_stats
*foo_get_stats(struct net_device *dev);
The net_device_stats structure must be filled by the driver
It contains fields such as rx_packets, tx_packets, rx_bytes, tx_bytes, rx_errors,
tx_errors, rx_dropped, tx_dropped, multicast, collisions, etc
September 7, 2017
Embedded Linux Network Device Driver
Development
44
45. Power Management
suspend() function should:
Call netif_device_detach()
Do the hardware dependent operations to suspend the devices (like disable
the clocks)
The resume() function should:
Do the hardware dependent operations (like enable the clocks)
Call netif_device_attach()
September 7, 2017
Embedded Linux Network Device Driver
Development
45
47. Outline
Introduction
Socket Buffers
Network Devices
Communication with Network Protocol
Communication with PHY
Buffer Management and Concurrency Issues
Network Throughput
Other Network Driver Considerations
Ethernet Driver vs WIFI Driver
September 7, 2017
Embedded Linux Network Device Driver
Development
47
48. Ethernet vs. WIFI
Ethernet
CSMA/CD
Frames are not acknowledged
WIFI
CSMA/CA
Frames are acknowledged
Uses WEP for enhanced security
2 modes of operation:
Ad-hoc
Infrastructure
September 7, 2017
Embedded Linux Network Device Driver
Development
48
49. Configuring WIFI Drivers
The Wireless Extensions project defines a generic Linux API to configure
WLAN device drivers in a device independent manner
It also provides a set of common tools to set and access information from
WLAN drivers
Individual drivers must implement support for Wireless Extensions to
connect themselves with the common interface and, hence, with the tools
September 7, 2017
Embedded Linux Network Device Driver
Development
49
50. Talking to WIFI Drivers
Using iwconfig utility
To glue your driver to iwconfig, you need to implement prescribed functions
corresponding to commands that set parameters such as ESSID and WEP keys
Using iwpriv utility
To use iwpriv over your driver, define private ioctls relevant to your hardware
and implement the corresponding handler functions
Using /proc/net/wireless to get driver statistics
For this, implement the get_wireless_stats() function in your driver, in addition
to the get_stats() function
WLAN drivers tie these three pieces of information inside a structure
called iw_handler_def, defined in include/net/iw_handler.h
September 7, 2017
Embedded Linux Network Device Driver
Development
50
51. Ethernet/WIFI Famous Device Drivers
September 7, 2017
Embedded Linux Network Device Driver
Development
51
Device(s) Driver Location in Kernel Tree
Intersil Prism2 WLAN Compact Flash Card /drivers/net/wireless/intersil/orinoco
Intel Pro/Wireless Mini PCI (and PCIe Mini) /drivers/net/wireless/intel/ipw2x00
Atmel WLAN USB /drivers/net/wireless/atmel
Intel PRO/1000 /drivers/net/ethernet/intel/e1000