Linux System Programming 学习笔记(二) 文件I/O

1.每一个Linux进程都有一个最大打开文件数，默认状况下，最大值是1024

文件描述符不只能够引用普通文件，也能够引用套接字socket，目录，管道(everything is a file)

默认状况下，子进程会得到其父进程文件表的完整拷贝

 

2.打开文件

open系统调用必须包含 O_RDONLY，O_WRONLY，O_RDWR 三种存取模式之一

注意 O_NONBLOCK模式

int fd = open(filename, O_WRONLY | O_CREAT | O_TRUNC, 0644) 
int fd = creat(filename, 0644)

3.读文件

read系统调用会有如下结果：

(1)返回值与请求数len相同，全部len字节都存储在buf内

 

(2)返回值小于请求数len，可是大于0。发生此种状况有不少缘由：

a.read系统调用中途被信号中断

b.read系统调用中途发生错误

c.可读字节数大于0，可是小于len

d.在读完len字节以前遇到EOF

 

(3)返回0，表示EOF

(4)调用被阻塞，由于当前没有可读，这种状况不会发生在非阻塞模式

(5)返回-1，errno设置为EINTR，表示在读任一字节以前就接收到信号

(6)返回-1，errno设置为EAGAIN，表示读操做被阻塞，由于当前并无可读字节，这只发生在 非阻塞模式

(7)返回-1，errno设置为 EINTR，EAGAIN以外的值，表示发生其余更严重的错误

 

读完全部字节：

size_t readn(int fd, void* buf, size_t len)
{
    size_t  tmp = len;
    ssize_t ret = 0;
    while (len != 0 && (ret = read(fd, buf, len)) != 0) {
        if (ret == -1) {
            if (errno == EINTR) {
                continue;
            }
            fprintf(stderr, "read error\n");
            break;
        }
        len -= ret;
        buf += ret;
    }
    return tmp - len;
}

非阻塞读：

有时咱们并不但愿当没有可读数据时read系统调用被阻塞，而是但愿调用能够当即返回，代表没有数据可读，这就是非阻塞I/O

 

4.写文件

write系统调用没有EOF，对于普通文件，write默认操做是所有写，除非是发生错误返回-1

对于其余文件就有可能发生部分写，最典型的是 网络编程中socket读写时，应该以下写：

size_t writen(int fd, void* buf, size_t len)
{
    ssize_t ret = 0;
    size_t  tmp = len;
    while (len != 0 && (ret = write(fd, buf, len)) != 0) {
        if (ret == -1) {
            if (errno == EINTR) {
                continue;
            }
            fprintf(stderr, "write error\n");
            break;
        }
        len -= ret;
        buf += ret;
    }
    return tmp - len;
}

追加模式能够确保文件的当前位置老是位于文件末尾，而且能够把文件偏移更新操做当作 原子操做，因此该模式对于多任务追加写很是有用

 

5.文件同步

当调用write时，内核从用户buffer拷贝数据到内核buffer，但并不必定是当即写到目的地，内核一般是执行一些检查，将数据从用户buffer拷贝到一个dirty buffer，后而内核收集全部这些dirty buffer( contain data newer than what is on disk)，最后才写回磁盘。

这种延迟写并无改变POSIX语义，反而能够提高读写性能

if a read is issued for just-written data that lives in a dirty buffer and is not yet on disk, the request will be satisfied from the buffer and not cause a read from the "stale" data on disk. so the read is satisfied from an in-memory cache without having to go to disk.

 

延迟写能够大幅提高性能，可是有时候须要控制写回磁盘的文件，这是须要确保文件同步

fsync系统调用确保fd关联的文件数据已经写回到磁盘

int ret = fsync(fd);

open调用时 O_SYNC标志表示 文件必须同步

int fd = open(file, O_WRONLY | O_SYNC);

O_SYNC致使I/O等待时间消耗巨大，通常地，须要确保文件写回到磁盘时咱们使用 fsync函数

 

6.文件定位

显式的文件定位函数：

a. 将文件偏移定位到1825

off_t ret = lseek(fd, (off_t)1825, SEEK_SET);

b. 将文件便宜定位到文件末尾处

off_t ret = lseek(fd, 0, SEEK_END);

c. 将文件偏移定位到文件开始处

off_t ret = lseek(fd, 0, SEEK_CUR)

 

文件定位是能够超出文件末尾的，此时对该文件写操做会填补0，造成空洞， 空洞是不占有物理磁盘空间的。

This implies that the total size of all files on a filesystem can add up to more than the physical size of the disk

 

7.截断文件

int ftruncate(int fd, off_t len);  

将给定文件截断为给定长度，这里的给定长度是能够小于文件大小，也能够大于文件大小(会形成空洞)

8.多路I/O

阻塞I/O：若是read系统调用时，文件(例如管道输入)没有可读数据，这时进程会一直阻塞等待，直到有可读数据。效率低下，不能同时进行多个文件读写操做

多路I/O能够容许程序并发地阻塞在多个文件上，而且当任一文件变为可读或可写的时候会立马接收到通知

Multiplexed I/O becomes the pivot point for the application，designed similarly to the following activity：
a. Multiplexed I/O ： Tell me when any of these file descriptors becomes ready for I/O
b. Nothing ready? Sleep until one or more file descriptors are ready.
c. Woken up ! What is ready?
d. Handle all file descriptors ready for I/O, without bolocking
e. Go back to step a

9.select

int select(int nfds, fd_set* readfds, fd_set* writefds, fd_set* exceptfds, struct timeval* timeout);
FD_CLR(int fd, fd_set* set);      // removes a fd from a given set
FD_ISSET(int fd, fd_set* set);   // test whether a fd is part of a given set
FD_SET(int fd, fd_set* set);      // adds a fd to a given set
FD_ZERO(int fd, fd_set* set);  // removes all fds from specified set. shoule be called before every invocation of select()

由于fd_set是静态分配的，系统有一个文件描述符的最大打开数 FD_SETSIZE，在Linux中，该值为 1024

#include <stdio.h>
#include <stdlib.h>
#include <sys/time.h>
#include <sys/types.h>
#include <unistd.h>

#define TIMEOUT 5     /* select timeout in seconds */
#define BUFLEN  1024  /* read buffer in bytes */

int main(int argc, char* argv[])
{
    struct timeval tv;
    tv.tv_sec = TIMEOUT;
    tv.tv_usec = 0;

    /* wait on stdin for input */
    fd_set readfds;
    FD_ZERO(&readfds);
    FD_SET(STDIN_FILENO, &readfds);

    int ret = select(STDIN_FILENO + 1, &readfds, NULL, NULL, &tv);
    if (ret == -1) {
        fprintf(stderr, "select error\n");
        return 1;
    } else if (!ret) {
        fprintf(stderr, "%d seconds elapsed.\n", TIMEOUT);
        return 0;
    }
    if (FD_ISSET(STDIN_FILENO, &readfds)) {
        char buf[BUFLEN + 1];
        int len = read(STDIN_FILENO, buf, BUFLEN);
        if (len == -1) {
            fprintf(stderr, "read error\n");
            return 1;
        }
        if (len != 0) {
            buf[BUFLEN] = '\0';
            fprintf(stdout, "read:%s\n", buf);
        }
        return 0;
    } else {
        fprintf(stderr, "This should not happen\n");
        return 1;
    }

}

 

10. poll

int poll(struct pollfd* fds, nfds_t  nfds, int timeout);

This is a program that uses poll() to check whether a read from stdin and a write to stdout will block

#include <unistd.h>
#include <poll.h>

#define TIMEOUT 5

int main(int argc, char* argv[])
{
    struct pollfd fds[2];

    /* watch stdin for input */
    fds[0].fd = STDIN_FILENO;
    fds[0].events = POLLIN;

    /* watch stdout for alibity to write */
    fds[1].fd = STDOUT_FILENO;
    fds[1].events = POLLOUT;

    int ret = poll(fds, 2, TIMEOUT * 1000);
    if (ret == -1) {
        fprintf(stderr, "poll error\n");
        return 1;
    }

    if (!ret) {
        fprintf(stdout, "%d seconds elapsed.\n", TIMEOUT);
        return 0;
    }

    if (fds[0].revents & POLLIN) {
        fprintf(stdout, "stdin is readable\n");
    }
    if (fds[1].revents & POLLOUT) {
        fprintf(stdout, "stdout is writable\n");
    }
    return 0;
}

poll vs select 

a. poll不须要用户计算并传递文件描述符参数(select中必须将该值设为最大描述符数加1)

b. select的fd_set是静态分配的，有一个最大文件数限制FD_SETSIZE，poll就没有这个限制，只须要建立一个合适大小的结构体数组

c. select移植性更好，支持select的unix更多

d. select支持更精细的timeout，poll只支持毫秒

 

11.内核实现

Linux内核主要由 virtual filesystem, page cache, page write-back 来支持有效且强大的I/O机制

(1) virtual filesystem

The virtual filesystem (also called a virtual file switch) is a mechanism of abstraction that allows the Linux kernel to call filesystem functions and manipulate filesystem data without knowing the specific type of filesystem being used.

So, a single system call can read any filesystem on any medium, All filesystems support the same concepts, the same interfaces, and the same calls

 

(2) page cache

The page cache is an in-memory store of recently accessed data from an on-disk filesystem.

Storing requested data in memory allows the kernel to fulfill subsequent requests for the same data  from memory, avoiding repeated disk access

The page cache exploits the concept of temporal locality, which says that a resource accessed at one point has a high probability of being accessed again in the near future

 

时间局部性：

The page cache is the first place that kernel looks for filesystem data. The first time any item of sata is read, it is transferred from the disk into the page cache, and is returned to the application from the cache. 

 

空间局部性：

The data is often referenced sequentially. The kernel implements page cache  readahead（预读）. Readahead is the act of reading extra data off the disk and into the page cache following each read request. In effect, reading a little bit ahead. 

 

(3) page write-back

When a process issues a write request, the data is copied into a buffer, and the buffer is marked dirty, denoting that the in-memory copy is newer than the on-disk copy.

Eventually, the dirty buffers need to be committed to disk, sync the on-disk files with the data in memory.