Google Drive Fuse filesystem

thejinx0r · November 16, 2014, 7:40pm

I've written my own filesystem to mount Google Drive.

For installation, see this page.

https://github.com/thejinx0r/node-gdrive-fuse/wiki/Installation

Feel free to post here and report issues on github.

What works:

File downloading / streaming. This will only download partial files (called chunks) and stitch them together as needed. The amount downloaded is user configurable. These chunks are also cached.
File uploading. Files will be cached by the filesystem and uploaded afterwards. File uploads will be resumed if the filesystem crashes. File uploads are also using the resumable upload API. Uploading large files (>20GB) will not be a problem.

What needs to be done:

Detecting remote file changes. Currently, it is unable to detect remote changes. It can only be aware of the changes made by the user. To manually update, delete the "folderTree.json" file from the data subfolder of the cache directory.

doughnet · November 16, 2014, 7:44pm

I've written my own filesystem to mount Google Drive.

https://github.com/thejinx0r/node-gdrive-fuse/wiki/Installation

Feel free to post here and report issues on github.

What works:

File downloading / streaming. This will only download partial files (called chunks) and stitch them together as needed. The amount downloaded is user configurable. These chunks are also cached.

File uploading. Files will be cached by the filesystem and uploaded afterwards. File uploads will be resumed if the filesystem crashes. File uploads are also using the resumable upload API. Uploading large files (>20GB) will not be a problem.

What needs to be done:

Detecting remote file changes. Currently, it is unable to detect remote changes. It can only be aware of the changes made by the user. To manually update, delete the "folderTree.json" file from the data subfolder of the cache directory.

awesome i'll try it out

what testing have you done to stress test it? (been able to scan a library of 10,000?)

where are chunks stored? if someone was wanting to run this on a small HDD (10GB) they wouldn't have much room for stuff to be stored locally.

thejinx0r · November 16, 2014, 7:50pm

My library is currently really small, but the Bitcasa one that I had written previously, was able to handle a library of >10k items.

The chunks are stored in a user defined location.

Everything is user configurable, including the amount of space being used.

doughnet · November 16, 2014, 7:54pm

Maybe provide some info/support explaining what prerequisites are needed (install git/node etc etc) and give some quick links on how to for them or just paste the command.

thejinx0r · November 16, 2014, 7:57pm

Maybe provide some info/support explaining what prerequisites are needed (install git/node etc etc) and give some quick links on how to for them or just paste the command.

It's in the link of the first page.

Here it is if you missed it:

https://github.com/thejinx0r/node-gdrive-fuse/wiki/Installation

It has pretty clear instructions.

doughnet · November 16, 2014, 8:00pm

simple questions you should put up:

need nodejs installed

need npm installed

how to install them? should i do them via apt or yum? should i compile from source? fresh debian 7 installation you can install nodejs via apt-get install nodejs, but it's not including npm so people will get stuck on that possibly.

hence, why i mentioned you should provide all the details necessary to properly and successfully use it

doughnet · November 16, 2014, 8:23pm

when doing 'npm install' there are tons of errors, assuming it installed (since you don't mention very many details or pictures of what to expect) I go on to the next step
while editing the 'config.json' file you skipped information regarding clientId & clientSecret. Also you do not provide details on what you should and shouldn't edit. Does the URL path for the redirect need to be valid? can it be an IP#? do the folders where the cache and mount will be located need to be created first before initial mount (you don't have that as a step so I assume it will be created)? Can the clientID & clientSecret contents be removed or be left how they are?
running 'coffee fs.coffee' output is the following and none of the steps you mentioned is initiated http://i.imgur.com/iinTm9m.png
if I do get to step 7.iii will this urlRedirect go to a my localhost? what if I am running something on port 80 already, will this conflict and not authenticate?
if I do get to step 7.iv this information that is provided through the urlRedirect is just for looking at or is it suppose to be added to config.json or does your software automatically add it?

This is why I believe it would be great to have everything documented. Should just take a couple minutes since I'm assuming you have done this procedure tons of times.

thejinx0r · November 16, 2014, 8:33pm

when doing 'npm install' there are tons of errors, assuming it installed (since you don't mention very many details or pictures of what to expect) I go on to the next step

while editing the 'config.json' file you skipped information regarding clientId & clientSecret. Also you do not provide details on what you should and shouldn't edit. Does the URL path for the redirect need to be valid? can it be an IP#? do the folders where the cache and mount will be located need to be created first before initial mount (you don't have that as a step so I assume it will be created)? Can the clientID & clientSecret contents be removed or be left how they are?

running 'coffee fs.coffee' output is the following and none of the steps you mentioned is initiated http://i.imgur.com/iinTm9m.png

if I do get to step 7.iii will this urlRedirect go to a my localhost? what if I am running something on port 80 already, will this conflict and not authenticate?

if I do get to step 7.iv this information that is provided through the urlRedirect is just for looking at or is it suppose to be added to config.json or does your software automatically add it?

This is why I believe it would be great to have everything documented. Should just take a couple minutes since I'm assuming you have done this procedure tons of times.

npm install should just work. some bugs still exist. thanks for reporting that node-cache was not present.

I've added an extra line on the wiki for configuring the config.json explicitly saying which parameters do not need to be edited.

The urlredirect will not interfere withanything.

doughnet · November 16, 2014, 8:37pm

npm install should just work. some bugs still exist. thanks for reporting that node-cache was not present.

I've added an extra line on the wiki for configuring the config.json explicitly saying which parameters do not need to be edited.

The urlredirect will not interfere withanything.

shouldn't assume npm will install automatically (which it doesn't) ... if you install nodejs from source code it will be, but if you install via apt-get debian repos it will not be installed. i was able to compile from scratch to get this done (should add this information to your INSTALLATION page to make it easier for users to get your application working ... if people can't get the first step to work easily ... doubtful they will progress on trying to do the rest
here is the npm install errors which it mentions it's an error with the application and fuse (not NPM) http://pastebin.com/hLwft687

thejinx0r · November 16, 2014, 8:40pm

shouldn't assume npm will install automatically (which it doesn't) ... if you install nodejs from source code it will be, but if you install via apt-get debian repos it will not be installed. i was able to compile from scratch to get this done (should add this information to your INSTALLATION page to make it easier for users to get your application working ... if people can't get the first step to work easily ... doubtful they will progress on trying to do the rest

here is the npm install errors which it mentions it's an error with the application and fuse (not NPM) http://pastebin.com/hLwft687

I don't see any errors.

It looks fine.

I've also pushed a fix so that "npm install" will also node cache.

Also, please feel free to edit the wiki. It will help others as well.

doughnet · November 16, 2014, 8:44pm

Lol weird, wonder why it didn't paste what I wanted it to ... below is the picture of the errors:

http://i.imgur.com/72ZhSzT.png

thejinx0r · November 16, 2014, 8:51pm

Lol weird, wonder why it didn't paste what I wanted it to ... below is the picture of the errors:

http://i.imgur.com/72ZhSzT.png

Do you have fuse installed?

If you do, then try to follow this issue:

https://github.com/thejinx0r/node-gdrive-fuse/issues/2

I don't have access to a linux machine right now, just a mac.

The problem is that the fuse library is not being found properly.

Simply locate the path fuse library, and use this environmental variable "LD_LIBRARY_PATH" instead of "LIBRARY_PATH" as reported in the github issue.

doughnet · November 16, 2014, 8:55pm

thank's for getting back to me. that didn't work.

http://i.imgur.com/VcBpKvq.png

PM me if you want me to setup a dev box for you to work with

thejinx0r · November 16, 2014, 8:57pm

are you using a mac or linux?

for linux: use LD_LIBRARY_PATH

for mac: use LIBRARY_PATH

doughnet · November 16, 2014, 8:58pm

are you using a mac or linux?

for linux: use LD_LIBRARY_PATH

for mac: use LIBRARY_PATH

yes, did both ways just to be sure

the path /usr/local/lib contains node_modules but nothing for fuse4js

martinbowling · November 16, 2014, 10:24pm

Error: Cannot find module 'googleapis'
  at Function.Module._resolveFilename (module.js:338:15)
  at Function.Module._load (module.js:280:25)
  at Module.require (module.js:364:17)
  at require (module.js:380:17)
  at Object. (/Users/martinbowling/Projects/node-gdrive-fuse/src/fs.coffee:1:10)
  at Object. (/Users/martinbowling/Projects/node-gdrive-fuse/src/fs.coffee:1:1)
  at Module._compile (module.js:456:26)

when I try and run coffee fs.coffee I get this error, should I just grab the nodejs google api libs from googles github?

thejinx0r · November 17, 2014, 1:16am

Error: Cannot find module 'googleapis'
  at Function.Module._resolveFilename (module.js:338:15)
  at Function.Module._load (module.js:280:25)
  at Module.require (module.js:364:17)
  at require (module.js:380:17)
  at Object. (/Users/martinbowling/Projects/node-gdrive-fuse/src/fs.coffee:1:10)
  at Object. (/Users/martinbowling/Projects/node-gdrive-fuse/src/fs.coffee:1:1)
  at Module._compile (module.js:456:26)

when I try and run coffee fs.coffee I get this error, should I just grab the nodejs google api libs from googles github?

Have you run "npm install"?

thejinx0r · November 17, 2014, 1:17am

yes, did both ways just to be sure

the path /usr/local/lib contains node_modules but nothing for fuse4js

Does it contain the fuse library?

martinbowling · November 17, 2014, 3:11am

ok looks like that might have failed didn't see the errors

gyp ERR! configure error

gyp ERR! stack Error: `gyp` failed with exit code: 1 gyp ERR! stack at ChildProcess.onCpExit (/usr/local/lib/node_modules/npm/node_modules/node-gyp/lib/configure.js:343:16) gyp ERR! stack at ChildProcess.emit (events.js:98:17) gyp ERR! stack at Process.ChildProcess._handle.onexit (child_process.js:810:12) gyp ERR! System Darwin 12.5.0 gyp ERR! command "node" "/usr/local/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild" gyp ERR! cwd /Users/martinbowling/Projects/node-gdrive-fuse/node_modules/fuse4js gyp ERR! node -v v0.10.33 gyp ERR! node-gyp -v v1.0.1 gyp ERR! not ok

npm ERR! fuse4js@0.1.9 install: node-gyp rebuild
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the fuse4js@0.1.9 install script.
npm ERR! This is most likely a problem with the fuse4js package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR! node-gyp rebuild
npm ERR! You can get their info via:
npm ERR! npm owner ls fuse4js
npm ERR! There is likely additional logging output above.
npm ERR! System Darwin 12.5.0
npm ERR! command “node” “/usr/local/bin/npm” “install”
npm ERR! cwd /Users/martinbowling/Projects/node-gdrive-fuse
npm ERR! node -v v0.10.33
npm ERR! npm -v 1.4.28
npm ERR! code ELIFECYCLE
npm ERR! not ok code 0

gyp: Call to 'pkg-config --libs-only-l fuse' returned exit status 127. while trying to load binding.gyp

thejinx0r · November 17, 2014, 4:05am

ok looks like that might have failed didn't see the errors

gyp ERR! configure error

gyp ERR! stack Error: `gyp` failed with exit code: 1 gyp ERR! stack at ChildProcess.onCpExit (/usr/local/lib/node_modules/npm/node_modules/node-gyp/lib/configure.js:343:16) gyp ERR! stack at ChildProcess.emit (events.js:98:17) gyp ERR! stack at Process.ChildProcess._handle.onexit (child_process.js:810:12) gyp ERR! System Darwin 12.5.0 gyp ERR! command "node" "/usr/local/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild" gyp ERR! cwd /Users/martinbowling/Projects/node-gdrive-fuse/node_modules/fuse4js gyp ERR! node -v v0.10.33 gyp ERR! node-gyp -v v1.0.1 gyp ERR! not ok

npm ERR! fuse4js@0.1.9 install: node-gyp rebuild
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the fuse4js@0.1.9 install script.
npm ERR! This is most likely a problem with the fuse4js package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR! node-gyp rebuild
npm ERR! You can get their info via:
npm ERR! npm owner ls fuse4js
npm ERR! There is likely additional logging output above.
npm ERR! System Darwin 12.5.0
npm ERR! command “node” “/usr/local/bin/npm” “install”
npm ERR! cwd /Users/martinbowling/Projects/node-gdrive-fuse
npm ERR! node -v v0.10.33
npm ERR! npm -v 1.4.28
npm ERR! code ELIFECYCLE
npm ERR! not ok code 0

gyp: Call to 'pkg-config --libs-only-l fuse' returned exit status 127. while trying to load binding.gyp

Which distro are you using? I think you are having the same problem as w00ds.

Topic		Replies	Views
Derailed topic that turned into GDrive API errors - ON HOLD but likely resolved Desktops & Laptops	104	1319	August 24, 2019
Google file stream y Plex ¿Alguien lo tiene en marcha? Español - Spanish	43	3285	January 8, 2020
Plex Cloud vs VPS - why we went with a VPS Apps & Creations server-linux	2173	43548	October 29, 2024
Cloud Sync not working properly - Sporadic, Glitchy General Discussions	102	323	December 21, 2019
Google Drive te banea por tiempo limitado usando Plex Cloud.... Español - Spanish	26	3079	January 8, 2020

Google Drive Fuse filesystem

Related topics