]>
Commit | Line | Data |
---|---|---|
1 | From 38c4f55850b118899c10a2811cd436b2d051303a Mon Sep 17 00:00:00 2001 | |
2 | From: Neil Brown <neilb@suse.de> | |
3 | Date: Thu, 30 Aug 2012 16:13:50 +0200 | |
4 | Subject: [PATCH 07/13] overlay: overlay filesystem documentation | |
5 | Patch-mainline: not yet | |
6 | ||
7 | Document the overlay filesystem. | |
8 | ||
9 | Signed-off-by: Miklos Szeredi <mszeredi@suse.cz> | |
10 | --- | |
11 | Documentation/filesystems/overlayfs.txt | 199 ++++++++++++++++++++++++++++++++ | |
12 | MAINTAINERS | 7 + | |
13 | 2 files changed, 206 insertions(+) | |
14 | create mode 100644 Documentation/filesystems/overlayfs.txt | |
15 | ||
16 | Index: linux-3.6-rc7-master/Documentation/filesystems/overlayfs.txt | |
17 | =================================================================== | |
18 | --- /dev/null 1970-01-01 00:00:00.000000000 +0000 | |
19 | +++ linux-3.6-rc7-master/Documentation/filesystems/overlayfs.txt 2012-09-28 13:36:58.000000000 +0200 | |
20 | @@ -0,0 +1,199 @@ | |
21 | +Written by: Neil Brown <neilb@suse.de> | |
22 | + | |
23 | +Overlay Filesystem | |
24 | +================== | |
25 | + | |
26 | +This document describes a prototype for a new approach to providing | |
27 | +overlay-filesystem functionality in Linux (sometimes referred to as | |
28 | +union-filesystems). An overlay-filesystem tries to present a | |
29 | +filesystem which is the result over overlaying one filesystem on top | |
30 | +of the other. | |
31 | + | |
32 | +The result will inevitably fail to look exactly like a normal | |
33 | +filesystem for various technical reasons. The expectation is that | |
34 | +many use cases will be able to ignore these differences. | |
35 | + | |
36 | +This approach is 'hybrid' because the objects that appear in the | |
37 | +filesystem do not all appear to belong to that filesystem. In many | |
38 | +cases an object accessed in the union will be indistinguishable | |
39 | +from accessing the corresponding object from the original filesystem. | |
40 | +This is most obvious from the 'st_dev' field returned by stat(2). | |
41 | + | |
42 | +While directories will report an st_dev from the overlay-filesystem, | |
43 | +all non-directory objects will report an st_dev from the lower or | |
44 | +upper filesystem that is providing the object. Similarly st_ino will | |
45 | +only be unique when combined with st_dev, and both of these can change | |
46 | +over the lifetime of a non-directory object. Many applications and | |
47 | +tools ignore these values and will not be affected. | |
48 | + | |
49 | +Upper and Lower | |
50 | +--------------- | |
51 | + | |
52 | +An overlay filesystem combines two filesystems - an 'upper' filesystem | |
53 | +and a 'lower' filesystem. When a name exists in both filesystems, the | |
54 | +object in the 'upper' filesystem is visible while the object in the | |
55 | +'lower' filesystem is either hidden or, in the case of directories, | |
56 | +merged with the 'upper' object. | |
57 | + | |
58 | +It would be more correct to refer to an upper and lower 'directory | |
59 | +tree' rather than 'filesystem' as it is quite possible for both | |
60 | +directory trees to be in the same filesystem and there is no | |
61 | +requirement that the root of a filesystem be given for either upper or | |
62 | +lower. | |
63 | + | |
64 | +The lower filesystem can be any filesystem supported by Linux and does | |
65 | +not need to be writable. The lower filesystem can even be another | |
66 | +overlayfs. The upper filesystem will normally be writable and if it | |
67 | +is it must support the creation of trusted.* extended attributes, and | |
68 | +must provide valid d_type in readdir responses, at least for symbolic | |
69 | +links - so NFS is not suitable. | |
70 | + | |
71 | +A read-only overlay of two read-only filesystems may use any | |
72 | +filesystem type. | |
73 | + | |
74 | +Directories | |
75 | +----------- | |
76 | + | |
77 | +Overlaying mainly involves directories. If a given name appears in both | |
78 | +upper and lower filesystems and refers to a non-directory in either, | |
79 | +then the lower object is hidden - the name refers only to the upper | |
80 | +object. | |
81 | + | |
82 | +Where both upper and lower objects are directories, a merged directory | |
83 | +is formed. | |
84 | + | |
85 | +At mount time, the two directories given as mount options are combined | |
86 | +into a merged directory: | |
87 | + | |
88 | + mount -t overlayfs overlayfs -olowerdir=/lower,upperdir=/upper /overlay | |
89 | + | |
90 | +Then whenever a lookup is requested in such a merged directory, the | |
91 | +lookup is performed in each actual directory and the combined result | |
92 | +is cached in the dentry belonging to the overlay filesystem. If both | |
93 | +actual lookups find directories, both are stored and a merged | |
94 | +directory is created, otherwise only one is stored: the upper if it | |
95 | +exists, else the lower. | |
96 | + | |
97 | +Only the lists of names from directories are merged. Other content | |
98 | +such as metadata and extended attributes are reported for the upper | |
99 | +directory only. These attributes of the lower directory are hidden. | |
100 | + | |
101 | +whiteouts and opaque directories | |
102 | +-------------------------------- | |
103 | + | |
104 | +In order to support rm and rmdir without changing the lower | |
105 | +filesystem, an overlay filesystem needs to record in the upper filesystem | |
106 | +that files have been removed. This is done using whiteouts and opaque | |
107 | +directories (non-directories are always opaque). | |
108 | + | |
109 | +The overlay filesystem uses extended attributes with a | |
110 | +"trusted.overlay." prefix to record these details. | |
111 | + | |
112 | +A whiteout is created as a symbolic link with target | |
113 | +"(overlay-whiteout)" and with xattr "trusted.overlay.whiteout" set to "y". | |
114 | +When a whiteout is found in the upper level of a merged directory, any | |
115 | +matching name in the lower level is ignored, and the whiteout itself | |
116 | +is also hidden. | |
117 | + | |
118 | +A directory is made opaque by setting the xattr "trusted.overlay.opaque" | |
119 | +to "y". Where the upper filesystem contains an opaque directory, any | |
120 | +directory in the lower filesystem with the same name is ignored. | |
121 | + | |
122 | +readdir | |
123 | +------- | |
124 | + | |
125 | +When a 'readdir' request is made on a merged directory, the upper and | |
126 | +lower directories are each read and the name lists merged in the | |
127 | +obvious way (upper is read first, then lower - entries that already | |
128 | +exist are not re-added). This merged name list is cached in the | |
129 | +'struct file' and so remains as long as the file is kept open. If the | |
130 | +directory is opened and read by two processes at the same time, they | |
131 | +will each have separate caches. A seekdir to the start of the | |
132 | +directory (offset 0) followed by a readdir will cause the cache to be | |
133 | +discarded and rebuilt. | |
134 | + | |
135 | +This means that changes to the merged directory do not appear while a | |
136 | +directory is being read. This is unlikely to be noticed by many | |
137 | +programs. | |
138 | + | |
139 | +seek offsets are assigned sequentially when the directories are read. | |
140 | +Thus if | |
141 | + - read part of a directory | |
142 | + - remember an offset, and close the directory | |
143 | + - re-open the directory some time later | |
144 | + - seek to the remembered offset | |
145 | + | |
146 | +there may be little correlation between the old and new locations in | |
147 | +the list of filenames, particularly if anything has changed in the | |
148 | +directory. | |
149 | + | |
150 | +Readdir on directories that are not merged is simply handled by the | |
151 | +underlying directory (upper or lower). | |
152 | + | |
153 | + | |
154 | +Non-directories | |
155 | +--------------- | |
156 | + | |
157 | +Objects that are not directories (files, symlinks, device-special | |
158 | +files etc.) are presented either from the upper or lower filesystem as | |
159 | +appropriate. When a file in the lower filesystem is accessed in a way | |
160 | +the requires write-access, such as opening for write access, changing | |
161 | +some metadata etc., the file is first copied from the lower filesystem | |
162 | +to the upper filesystem (copy_up). Note that creating a hard-link | |
163 | +also requires copy_up, though of course creation of a symlink does | |
164 | +not. | |
165 | + | |
166 | +The copy_up may turn out to be unnecessary, for example if the file is | |
167 | +opened for read-write but the data is not modified. | |
168 | + | |
169 | +The copy_up process first makes sure that the containing directory | |
170 | +exists in the upper filesystem - creating it and any parents as | |
171 | +necessary. It then creates the object with the same metadata (owner, | |
172 | +mode, mtime, symlink-target etc.) and then if the object is a file, the | |
173 | +data is copied from the lower to the upper filesystem. Finally any | |
174 | +extended attributes are copied up. | |
175 | + | |
176 | +Once the copy_up is complete, the overlay filesystem simply | |
177 | +provides direct access to the newly created file in the upper | |
178 | +filesystem - future operations on the file are barely noticed by the | |
179 | +overlay filesystem (though an operation on the name of the file such as | |
180 | +rename or unlink will of course be noticed and handled). | |
181 | + | |
182 | + | |
183 | +Non-standard behavior | |
184 | +--------------------- | |
185 | + | |
186 | +The copy_up operation essentially creates a new, identical file and | |
187 | +moves it over to the old name. The new file may be on a different | |
188 | +filesystem, so both st_dev and st_ino of the file may change. | |
189 | + | |
190 | +Any open files referring to this inode will access the old data and | |
191 | +metadata. Similarly any file locks obtained before copy_up will not | |
192 | +apply to the copied up file. | |
193 | + | |
194 | +On a file opened with O_RDONLY fchmod(2), fchown(2), futimesat(2) and | |
195 | +fsetxattr(2) will fail with EROFS. | |
196 | + | |
197 | +If a file with multiple hard links is copied up, then this will | |
198 | +"break" the link. Changes will not be propagated to other names | |
199 | +referring to the same inode. | |
200 | + | |
201 | +Symlinks in /proc/PID/ and /proc/PID/fd which point to a non-directory | |
202 | +object in overlayfs will not contain valid absolute paths, only | |
203 | +relative paths leading up to the filesystem's root. This will be | |
204 | +fixed in the future. | |
205 | + | |
206 | +Some operations are not atomic, for example a crash during copy_up or | |
207 | +rename will leave the filesystem in an inconsistent state. This will | |
208 | +be addressed in the future. | |
209 | + | |
210 | +Changes to underlying filesystems | |
211 | +--------------------------------- | |
212 | + | |
213 | +Offline changes, when the overlay is not mounted, are allowed to either | |
214 | +the upper or the lower trees. | |
215 | + | |
216 | +Changes to the underlying filesystems while part of a mounted overlay | |
217 | +filesystem are not allowed. If the underlying filesystem is changed, | |
218 | +the behavior of the overlay is undefined, though it will not result in | |
219 | +a crash or deadlock. | |
220 | Index: linux-3.6-rc7-master/MAINTAINERS | |
221 | =================================================================== | |
222 | --- linux-3.6-rc7-master.orig/MAINTAINERS 2012-09-24 03:10:57.000000000 +0200 | |
223 | +++ linux-3.6-rc7-master/MAINTAINERS 2012-09-28 13:36:58.000000000 +0200 | |
224 | @@ -5104,6 +5104,13 @@ F: drivers/scsi/osd/ | |
225 | F: include/scsi/osd_* | |
226 | F: fs/exofs/ | |
227 | ||
228 | +OVERLAYFS FILESYSTEM | |
229 | +M: Miklos Szeredi <miklos@szeredi.hu> | |
230 | +L: linux-fsdevel@vger.kernel.org | |
231 | +S: Supported | |
232 | +F: fs/overlayfs/* | |
233 | +F: Documentation/filesystems/overlayfs.txt | |
234 | + | |
235 | P54 WIRELESS DRIVER | |
236 | M: Christian Lamparter <chunkeey@googlemail.com> | |
237 | L: linux-wireless@vger.kernel.org |