Duplicate locations for a single file with Hard Links

General Discussions
, , ,
1

INTRODUCTION (Feel free to skip)

So I’ve been putting a lot more work into my Plex library under Pandemic isolation, & I’ve started cleaning things up, trying to remove duplicates & I realized I have a bunch of stuff in two places. For me it’s mostly things like Christmas movies that are part of a series that I have 2 copies of the file, 1 as a movie, & another as an episode of the show. So last night I decided to get rid of the duplicates by replacing them with Hard Links. & I went to search the forums to see if anyone had any tips or best practices for Hard Links or if there was any reason to use Symbolic Links instead. I was surprised to note that none existed except some issues with Symbolic Links so I decided to start a thread, share my practices, & hopefully help others as well as encourage anyone with different methods to suggest them. So on that note let’s begin
NOTE: While I am using Windows 10 & all the methods outlined are using Windows 10, the same methods should work on all NTFS drives in Windows 7, Windows 8, & even Vista & XP.
ALSO NOTE: These practices should also work in Linux & Unix (Including Mac) systems, but will require knowledge of the Hard Link commands for those operating systems.

  • So 1ˢᵗ, I chose to use Hard Links & I believe they work better unless you are doing 1 of the following
  1. Linking to a different drive or partition or to a network location
  2. Like things breaking on you
  • 2ⁿᵈ I’m well accustomed to using Hard Links

So lets go over a couple requirements before we begin

  • You must be using NTFS (HFS on Mac or EXT4/EXT3/EXT2 on Linux)
  • Your location has to exist on the same file structure table, both will be essentially pointing to the same spot on the disk so they both have to be able to see that spot. I get more into that in a bit.

Now to dispel some concerns you might have

  • Both files do NOT need to have the same name
  • After being created the files CAN be moved or renamed without worrying about breaking the link
  • Deleting 1 does NOT delete the other. The file will continue to exist until ALL links are deleted
  • If you edit a Hard Linked file those edits will be reflected on all versions
  • There is no risk of Infinite Loops from Hard Linked files, Infinite loops can only occur when I directory nests within a parent to itself. As Hard Links are not relative pointers that cannot happen. Plus, since Hard Links are limited to files & not possible for folders at all this issue can never occur. The Link Shell Extension does give the ability to create Junctions which could lead to that problem if used carelessly. Junctions, like Symbolic Links are relative links to folder, used natively within Windows & other Operating Systems so that common files can exist across profiles without taking extra disk space.

ABOUT HARD LINKS: What Are They? How Do They Work? (Feel free to skip)

A Hard Link is often referred to as being like a shortcut, but it actually has absolutely no similarities to a shortcut. Every Hard Link is the actual file. Here’s how it works. When you create a file in the directory P:\Files\MyMovie.mkv nothing actually exists in any folder, folders are illusions. that file is known to the Drive by an Inode which is like it’s ID Tag. When you create that file an Inode is created for it telling the File System which blocks of the disk contain the information for that file. Lets say your file’s Inode is 1234abcDE in the File System you have a table telling it to show Inode 1234abcDE in the Directory P:\Files\ under the name MyMovie.mkv. When you move the file to Directory P:\Archive\When I Was Young\ nothing is actually moved on the disk, the information on the table for the Inode 1234abcDE as just been updated. If you make a copy of the file a new Inode, lets say 34dS87LLg is created & though the files may be identical, they are separate files.
So when you create a Hard Link you are just adding a 2nd Entry to that Inode. This is different from a Symbolic Link which is essentially a Shortcut in disguise because when you open a SymLink the system actually opens whatever is at the location that was created. so if you rename a different file to the name that was SymLinked it will open that file, just like how a shortcut does. Symbolic Links are a bit different from Shortcuts & act more like Aliases do on a Mac, but that’s a different topic.
Now, you know how when you delete a file it can be recovered because the data is still there? What happens is that when you delete a file it removes that Inode reference from the File Table. But the data still exists in those blocks & the Inode still exists in the table. If you’ve ever done file recovery where you can recover a file but not it’s name that means that the Inode was lost but the data blocks weren’t overwritten yet. It rarely happens, but it is possible. With a Hard Link when you delete 1 file it removes that entry from the Inode but leaves the other until all versions of that file have been removed.

Now Hard Links are created in the command prompt, with a very simple single line of code, but if you are uncomfortable in the command prompt or, as I discovered, if you are doing a lot of them, there is a shell extension called Link Shell Extension that makes it super easy. Before Quarantine I didn’t create them enough to try it, but since I have I love the extension. I’ll include a link in the 1st comment just in case there’s an issue with it being included.

So to create a Hard Link

mklink /h "Path of NEW file being created" "Path of ORIGINAL file"

So if I was taking a file I already had in P:\Files\MyMovie.mkv & making a Hard Link in P:\Archive\When I Was Young\EmbarassingMovie.mkv I would use the following command

mklink /h "P:\Archive\When I Was Young\EmbarassingMovie.mkv" "P:\Files\MyMovie.mkv"

Note that the file names do NOT have to be the same. Also note that you can edit the names in the future & it won’t affect the other or break the link.

Now with the Link Shell Extension you can Right click & select Pick Link Source & then Right click where you want it & select Drop As... > Hardlink or Right Drag & choose Drop Here... > Hardlink the same as you would a copy function.

Now to do this with Plex a few criteria are required

  • You CANNOT create hard links within the same library. I guess Plex must use the Inode for the file because if you try to, say, make a special show up at the end of the season AND in the Specials season Every time Plex refreshes the Metadata it will move it to the other one. You also cannot have a Special crossover of 2 shows show up in both shows in the same library. I tried many different ways, & was unable to find one that worked.
  • Library folders MUST exist on the same drive
  • Watch status from both files will be the same. If you are watching 1 file & stop halfway through the other will show as half watched. If you mark one as Unplayed the other will be marked as Unplayed as well. This doesn’t apply if you have the other entry in a Movie library. I’m not sure exactly why, but even if the other library is using a different scanner & different Metadata Agent or is matched to a different show it carries over. But it doesn’t from a Show to a Movie

So for my 1ˢᵗ example I’m going to use a Holiday Movie Library to add Christmas Specials of Doctor Who as Movies

  • My Doctor Who Episodes exist in P:\TV\Doctor Who (2005)\
  • My TV Library Points to P:\TV\
  • My Holiday Library points to P:\Holiday & I have all Christmas movies in the P:\Holiday\Christmas subfolder (I have not had issues but I know some people may have issues with Special Features when in a nested folder)

I’ll use one of my newest additions for this example
1ˢᵗ I create the folder for the Hard Linked movie at P:\Holiday\Christmas\Doctor Who A Christmas Carol (2010)
Then I create the Hard Link with the following command

mklink /h "P:\Holiday\Christmas\Doctor Who A Christmas Carol (2010)\Doctor Who A Christmas Carol (2010).mp4" "P:\TV\Doctor Who (2005)\Season 5 - Smith\Matt Smith S05E14 - A Christmas Carol.mp4"

That’s it. Done. Easy
You’ll note that I used a different name for the 2nd file. I could also have used the same name & then renamed it.

For the 2ⁿᵈ example I’ll use the Link Shell Extension
Same base starting points

  • My Doctor Who Episodes exist in P:\TV\Doctor Who (2005)\
  • My TV Library Points to P:\TV\
  • My Holiday Library points to P:\Holiday & I have all Christmas movies in the P:\Holiday\Christmas subfolder

1ˢᵗ I create the folder for the Hard Linked movie at P:\Holiday\Christmas\Doctor Who The Runaway Bride (2006)
Then I create the Hard Link by right-dragging from P:\TV\Doctor Who (2005)\Season 2 - Tennant\ the file Tennant S02E14 - The Runaway Bride.mp4 to P:\Holiday\Christmas\Doctor Who The Runaway Bride (2006)
one the Pop-Up Menu I select Drop Here... > Hardlink
I now rename P:\Holiday\Christmas\Doctor Who The Runaway Bride (2006)\Tennant S02E14 - The Runaway Bride.mp4 to P:\Holiday\Christmas\Doctor Who The Runaway Bride (2006)\Doctor Who The Runaway Bride (2006).mp4

It sounds more complicated, but it’s actually easier.

For the 3ʳᵈ example I’ll use Link Shell Extension but I’ll create the original file in the same directory & then move it.
Same base starting points

  • My Doctor Who Episodes exist in P:\TV\Doctor Who (2005)\
  • My TV Library Points to P:\TV\
  • My Holiday Library points to P:\Holiday & I have all Christmas movies in the P:\Holiday\Christmas subfolder

In the folder P:\TV\Doctor Who (2005)\Season 1 - Eccleston\ I right-drag the file Eccleston S01E14 - The Christmas Invasion.mp4 & release on another part of the same folder
On the PopUp I select Drop Here... > Hardlink
I rename Eccleston S01E14 - The Christmas Invasion - Hardlink.mp4 to Doctor Who The Christmas Invasion (2005).mp4
I then create the folder P:\Holiday\Christmas\Doctor Who The Christmas Invasion (2005)\
I drag Doctor Who The Christmas Invasion (2005).mp4 from P:\TV\Doctor Who (2005)\Season 1 - Eccleston\ to P:\Holiday\Christmas\Doctor Who The Christmas Invasion (2005)\

Every file plays without issue.

It’s a very simple process, practically foolproof, but I know a lot of people are afraid to mess with things they don’t understand like Hard Links So I hope this helps make that easier

1 Like
2

Link to Link Shell Extension I believe it to be safe, but will always advise doing a virus scan of the file before installing

4

This is actually completely untrue. Hard Links only point to files, not folders, there is no way to create an infinite loop with Hard Link files. The equivalent for a directory would be a Junction where that possibility does indeed exist & I understand people being afraid of Junctions

6

An example of what happens when you try to create a Hard Link within the same series in the same library. The same happens if you do it across shows in the same library but this shows it happening. You’ll notice that at the start it shows the season labeled S1: Director's Cut as having 14 episodes. That’s where the original file is located. I created a Hard Link to the special in both the Specials season as well as the Season 1 season. When I did a Refresh Metadata you see Specials increase to 2 Episodes & S1: Director's Cut decrease to 13, then you see Season 1 increase to 27 & Specials decrease back to 1.
Every time metadata is refreshed the same process happens with it ending on a different one.

That’s why the Hard Links cannot exist within the same Library

7

Hard links are neat. They’re a double-edged sword, and like other useful tools, they can surprise you.

It’s safe to rename and delete individual links. And because they’re just additional names for the underlying file, they don’t use additional space.

But there really is only one file. Regardless of which link was opened, if the file is changed, all links reflect that change.

That’s a big violation of the Principle Of Least Astonishment, because it’s very different from how files usually work.

It’s damned hard to remember that THIS file is a hard link, but THAT file is a copy. So it’s safe to edit THAT file, but not THIS one.

That’s not to say that double-edged swords are bad. Just that they’re sharp.

They often confuse “Space Used” and “Space on Disk” calculations.

Backup utilities may not understand them. (Or conversely, backup systems may be built on them.)

It’s OK to delete a hard link. But if you Secure Delete a hard link, what has happened to the contents?

8

In windows it’s a little easier because all Hardlinks show a little Red Arrow on the corner, similar to the shortcut Blue arrow only red. When you create a Hard Link both the original as well as the new one get that arrow so you always know.
I do agree that it can be an issue, but in this use case I feel it is not. If you want to delete all Hard Links to a file you can go to the Properties of the file & under the Link Properties tab it will list all locations the file is in. If you delete all but one that arrow will disappear from the remaining file.

9

I think the overlay icon is a feature of the Link Shell Extension … which I agree makes that extension even more valuable.

10

I guess you’re right. I’ve been using Hard Links for a long time & only have been using the Extension for just over a month, but it has been so integrated into how I do things I forgot that was part of it until I just checked on a different system. Good to note

11

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.