File Storage

[adrianed]

Hi Folks,
After reading the posts from pegwell and Drake I decided to comment about file storage I read about recently
Many of us store backups etc on usb drives or other ssd drives
I read some questions and answers online this week on a computer forum a question was asked if it was a good idea to store items on ssd hdd or usb drives and most of the replies were a resounding no
Most replies say that those types of storage devices when they are left out of continuous service ie stored lose information
So it is not a matter of having a safe period of time, if they lose information then it can start any time from the beginning dropping bit by bit losing information, of course there will be many variables
It has reduced my confidence in these products for storing backups etc
For anyone who is interested it is worth reading a few things about this on the internet, sorry I didn't keep any links for reference
Adrian

[Arnak]

As an X IT professional, I agree with Adrian, any magnetic media is going to lose reliability over time.

The normal method for very large data sets is to back up to a tape drive using the Grandfather, Father, Son system.

Overwriting the oldest tape in rotation so that you always have 3 copies,1 for each day, then take the latest tape at month's end and put that in safe fireproof storage.

I have successfully restored a machine from a year-old tape but I wouldn't want to rely on that!

For smaller datasets, perhaps burning to a CD or BlueRay disk on a regular basis might be the best way forward, at least it is cheap.

I use a cheap old pc linked to the Tyros 4 and regularly back up to that then burn a CD with the data when I have made changes that I would not like to have to make again.

Martin

[BogdanH]

When we talk about data backup for archiving purposes, then we should always define the time span: are we talking about a month, a year, 10 years,..? At the same time, we should also define the term "safe" backup and importance of data we wish to preserve.
I say that, because at the end, someone might suggest we should have one backup copy at home, one at local bank and one deep in Switzerland mountains -we can see similar suggestions from "smarties" on some forums.
But let's be reasonable: our life doesn't depend on our data and nobody wants to be a backup slave (by making backup duplicates every week).
My experience (from four decades) is, it's very rare that data is lost because of hardware.. usually it happens because we're not careful: we erase data by mistake, we can't find the data (which USB stick was it?), fresh OS installation was made for whatever reason, etc.

To keep post short, my recommendation is simple and effective: get a NAS (2-bay minimum, don't use RAID 0 mode). For example, a 2-bay NAS with two 2TB HDD's (=2TB capacity) is about 350€. And beauty of that is, if it happens that one disk fails, your data is still preserved and so you have time to replace it.
Point being: you will always know where your data is -and if you loose it, you know it's your fault.

Just my 2c,
Bogdan

[adrianed]

Interesting replies Bogdan and Martin,
Over the past 25 years all of my breakdowns have been due to Windows failures not the regular disk drives
It hasn't been happening so much in recent years
So that being I mostly reinstall windows from windows media online using another computer but we need to keep a backup separate of all the other things programs and files.
Programs can be reinstalled again but it takes time and can we remember all the things we had
My main reason for my post was because I know players have made it known that they backup the music keyboard onto usb drives and store them in case of a keyboard breakdown
They may be stored for years and when they are needed they might not work, this would be a major shock
This is why I mentioned the backup problem just to make it known
Adrian

[BogdanH]

hello Adrian,

...
My main reason for my post was because I know players have made it known that they backup the music keyboard onto usb drives and store them in case of a keyboard breakdown..

You mean USB sticks I assume. USB stick is a temporary storage media in first place (i.e. to copy data between two devices which are not connected) and shouldn't be seen as backup solution -we can't expect that for 10€.

If at all, then I would use portable USB disk drive (which has 2.5" HDD inside -same as in many laptops). Such drive can hold data for few years (I used it in the past). It's relative cheap (50-60€) archiving solution, however it must be handled carefully enough (HDD is a precise/sensitive mechanical device).
Nowadays a bit more expensive SSD's are replacing HDD: they're faster (depending on USB connection) and they are quite robust (i.e. no problem if dropped on floor).
No, portable drive is still not a "safe" backup, but is still better than USB stick, in my opinion.

As I mentioned, the only safe backup (if we exclude major earthquake, fire, flood, etc.) is NAS in RAID 1 (=mirror) mode. Here data is stored on two drives at the same time, where NAS is checking all the time if both drives actually contain the same data. And if difference is detected, user gets a message telling which drive is failing and that it should be replaced.
Downside is, it's not really portable and is not that cheap. But there are a lot of benefits, where I mention a few:
- data is very secure,
- high transfer speed,
- every device connected to router (wired or wi-fi) can read/save the data from/to NAS,
- more than one user can read/save data simultaneously,
- admin defines who can read/save data and where data is stored. Means, every user can have it's own reserved storage space (which can be very useful if other family members are also storing data on NAS).

I hope that is informative for some.
Bogdan

[adrianed]

Yes Bogdan, I did mean usb stick, small variations in the language
I might try a NAS settup but usb sticks seemed like a good quick method of being safe, they and ssd storage devices have the same problem of leakage.
That was the point of my post
Adrian

[BogdanH]

All clear Adrian. Everyone should use whatever works for him.. reality is, USB sticks are useful and there's no reason to avoid them (looking at USB sticks on my desk ).

Greetings,
Bogdan

[Toril S]

This whole discussion scares the livintg sh.... out of me! I have thousands of files, music, pictures, text, my whole LIFE is in those files! Two of my USB sticks stopped functioning, but I fortunately had backups. File storing business is a nightmare!

[BogdanH]

hi Toril,
Actually it's interesting the way we think about all this.. We are aware of importance of our data (pictures, movies, personal documents, software that we bough, passwords, music stuff, etc.), still we pay almost no attention on how to preserve all that relative securely. Now and then we buy new PC/laptop, we buy new monitor or upgrade other devices, new phone maybe -all things that are replaceable anytime. Our personal data is not replaceable and still we do nothing about.

I know I repeat myself... but when we decided to have a PC/laptop (that's not only for internet browsing), we should at the same time buy a NAS device -it's the only long term solution. How to say.. NAS should be a part of our equipment (like modem and router is).

Just sharing my thoughts,
Bogdan

[adrianed]

You can say that again Toril
Wife and I also have all lifelong photos and music stored on hard drives, I shall have to take a look at everything
I am going take a look at this NAS idea that Bogdan suggests, I am sure 2tb won't be enough though
I suppose then we shall find something bad about that after setting it up and using it for years, how about back to the trusty shoe box
Adrian

[dlepera]

Folks,

  Let me share my thoughts and experiences on backups and data integrity. Bogdan is correct on the Raid Disk capabilities and fail safe aspect. But for us using them for the purpose of ensuring that our music files are protected, there are choices like Solid State Disk devices and paying for cloud storage for some of your needs. Laptops well they are volatile if you have an HHD and better off with SSD. Not talking about some of the **** operating system failures and bugs of course. 
       
   I am a storage backup pig in that I have over 12TB of personal data,  emails from the Windows back in the early 90s(with old pgms to support them) , family pictures and movies from the 80's residing on yes one HDD disk pulled out of 1987 HP laptop in an external case and formatted as an external hard drive and a couple EHDs. I have Toshiba external hard drive(many years old), WD external hard drives & cloud media drive and over 30 USBs of all storage capacities and speeds.  In addition to that I have backed them all up to WD solid state multiple 4TB Passports All organized and labeled to cover off all my data stored on all the devices that I have mentioned above for contingency backup.

    I have in over 33 years had only had 5 usb jump drive recent failures, 2 Lexar 16gb(4yrs old) and 3 Sandisk(1 32gb and 2 64gb) with these USBs being only 2 years old.

    I do every year plug in my old devices to do an integrity check and keeping them in use for about 2-3 hours and no failure yet(fingers crossed). An no drive deterioration. 

   The external hard drives are all disconnected when they are not in use. Note that all spinning disk have a predictable spin life span determined by the manufacture, not to mention that heat generated by the device spinning contributes to the life and functionality of the disks and SSDs can and do generate some little heat depending on their use(active programs accessing them for what ever reason  For me the USBs when not in use are removed from the devices that they connect to and I store them in a nice cool dark place and none are full more than 90%, not that darkness makes a difference. I minimize the plugging and unplugging of the precious devices and take clones of the ones that I want to frequently use so if anything should go array, I can quickly restore.

    My laptops all 5-7 years old all have SSDs and in my desktop I have installed an SSD with the HDD being my backup bootable drive along with addition 512gb storage cloning my SSD Desktop content. 
   
   As for my keyboard, if some of you have read one or two of my posts, I have stated that I had 3 USBs connected to my keyboard by a 4-port USB HUB all the time and all are clones of each other.  I do my saves in triplicate with no know issues for me, but Chris in one of my posts indicated that a longer EXCUTION/CHECKING stage process of the keyboard is being bogged down by these multiple drives and he is correct. So since then I keep one connected at all times and toggle the others on and off as I see fit when preserving my Registration only. So when it is necessary to do a backup of my USB, I fire up the other two on my keyboard and do the file delete of the old and copy the updated Registration folder to the backups. This is now quicker for me than waiting on the keyboard to do it's execution/checking phase.  I have these clone copies USBs since I understand that the true path is required for my Registrations to work if they are using styles, voices, pads, etcs outside of the resident inboard stuff since Registration know of the Root locations and moving things around or bringing them in from other sources to rebuild would hinder the workings of the Registrations. Done that and got burned in my earlier years on this keyboard. 

        Crazy, eh?  Not as crazy as loosing data and going trough what Toril feared. Toril, you stated "File storing business is a nightmare!" I have about 1.5 million files of all sorts in many meaningful folders and easy to manage because most of them are static. Over time(your storage evolution) you have I am sure created meaningful static folders as well. Many are static and used as references viewing pleasure and then you have the dynamic folder and files, those are can be easily manageable and what I consider precious. In case of a USB failure as was your case, had you had a clone, a restore in minutes is all it would have taken by simply using your cloned USB, unless you need to create a backup of the failing drive onto a new one to complete the "crazy" data integrity process that I have.

   Here is another example of what I do on my phone.. I have 1.5TB on my phone, 512gb internal and 1TB SD card. So every month for example, I Rename my Camera, Download and any critical folder to say Pics 2023 Jan, Down 2023 Jan, ScreenShot 2023 Jan, etc and move from Internal to SD storage. Yes I also copy to my WD passport to backup my SD card any time a new folder is created or if I feel I have "precious" data that I do not want to loose.  This naming conventions gives me a nice sort and at a glance I know exactly what is contained in them and they become static and require no more attention to maintain and no management nightmare.
                 
   While this is long winded it has worked for me for decades and workload to maintain and manage is minimal with a very high level of data integrity, simplified restoration and most valuable of all peace of mind.
May be expensive at first if you have to set this up from scratch but one does not have to go into overload mode like me. Keep in mind what is the value of your data, how much time did you put into it to create, can it ever be recreated, if so how much would it cost and how long would it take if you can ever get anything back.  It's like car insurance, We pay a big premium and hope we never have to use it, but if you ever do, at least you are protected.


Hope this helps.
                               Regards.     dom

           

[BogdanH]

..
I am going take a look at this NAS idea that Bogdan suggests, I am sure 2tb won't be enough though
I suppose then we shall find something bad about that after setting it up and using it for years,..

I mentioned 2TB because the price is almost the same as for lower capacities (i.e. 1TB). Everyone should determine the storage volume needed for himself, of course. Being at that, we also need to "optimize" how we save our data, otherwise we might end up with 2000€ 8-bay 60TB NAS. For example, if talking about music, saving it as wav file isn't really a smart idea: if we compress wav files, we can save ~40% of storage space and if we convert to FLAC, we save up to 60%. Of course, size of files that are already compressed (mp3, flac, jpg, zip, etc.) cannot be reduced any further.
At the same time we also need to have our files organized: myPhotos, myMusic, myVideos, mySoftware, myKeyboard, etc. That way we avoid having the same files at multiple places.

From my experience (I'm not a backup slave -I want it simple!), the best approach to determine the needed storage space is, that we start with double space of data size that we currently have. For example, if our current amount of files makes about 1.8TB, then we buy a 4TB system. That means, for 2-bay NAS, we need two 4TB drives. Why not more? Because experience shows, that once we reach our average amount of data, that amount doesn't increase that fast as it did in past. That is, we needed years to come up to 1.8TB and many more years will be needed until the amount of our data duplicates.
All that by assuming, that we will also delete obsolete files occasionally (for example, manuals of keyboards we no longer have, software that was replaced by newer version, etc.).

For those not familiar with NAS... It's not some "techie" device and it requires no special knowledge. Actually it's quite primitive device, which main purpose is to store files. Basically you use it the same ways as you use USB stick (although much faster) and it can do much more that that -but that's not relevant in this topic.
In short, once you have it, you'll wonder how could you be without it -enough said 

Bogdan

[dlepera]

Hello Bogdan.

  You are absolutely correct in what you say. The main point is that one has to really determine their storage requirement(key factor in all this) and what best fits their budget keeping in mind that storage prices keep dropping while capacity increases.

   For those that don't know what NAS is, here is a link that might give you some idea.  I would suggest though if possible to spend the little extra on an SSD to use in it, but keep in mind that a SATA disk by SEAGATE is very reliable as you know from these old computers with them and still going strong after many years.

https://www.google.com/search?q=what+is+2-bay+NAS&rlz=1C1CHBD_enCA839CA839&oq=what+is+2-bay+NAS&aqs=chrome..69i57j0i22i30i625j0i390l2.5823j0j7&sourceid=chrome&ie=UTF-8#fpstate=ive&vld=cid:b09a79e9,vid:OcJXFmmLYZA

Now with this all said I suspect that anyone wanting to only focus on preserving their music data and the fact that they may and will use Regits, voices, pads and styles from different folders on their USBs drive(s), they would need to think exactly how to preserve that data with minimal manipulation and high degree of integrity.

    Keep a copy of the the components in separate designated folders on any type of storage device. If downloading from the forum, I suggest that you preserve the same file structure, much the same as what one sees on the forum in Collections for example. This would give one the opportunity to rebuild in worse case scenarios from any storage device of choice which contains your backup. Also it ensures that if you downloaded anything specific from the forum posts that people have put links to their cloud storage content then later removed the content, that you will always have it in your possession to go back to.

   Then the important one make a clone of your active USB on your computer with a simple copy of everything(select all) to a new comparable USB. Once completed of course test the new USB with it only being connected to the keyboard to ensure that everything works as expected.

   This would be cheap, simple and reliable. 

           dom

[Joe H]

I personally have not had any of the experiences noted above.  But seems to me if you are uncertain about magnetic devices (which includes the cloud... it's just a hard drive somewhere in the world) then you might consider backing up on a DVD disk.

Joe H

[adrianed]

Joe,
If we do partial backups to look after daily or perhaps even monthly changes we are going to have a lot of disks.
The good thing about Hard drives is that we can delete old backups and write over them
It is possibly a decent idea to have a master backup on a DVD and do incremental backup on a hdd fairly often
I am now hesitant to say anything for certain because there doesn't seem to be an easy safe idea available for casual use
Dom has sent us a link to read regarding NAS I haven't read it yet but thank you dom
Adrian

[dlepera]

Adrian,
  You don't want to complicate a simple backup strategy or even complicate it with DVD.  DVDs are going the way of Beta, VHS, CDs, I never told you that I converted over 1500 movies, family videos and music which I had on DVDs and CDs to electronic media.  This is the way of the future. Look at any PC , try to find one with a DVD, CD reader.  In fact not sure where you are but try finding a Disk player that can even read some of them. The Codecs will prevent many from being read on a lot of media players. In fact some play on my old WD media player but not on my Samsung and disk recorders are even harder to find.  So, my advice is to make things simple and stick with current technology that "works",   USBs, Micro SDs, NAS, WD, Sandisk storage, etc.
         While incremental backups are great for server base environment, you may have to be careful with those related to your keyboard. Again, the root path for a registration is important so my understanding that the best backup is an all or nothing approach to keep all the components of the musical file in tack.

  As for the NAS link, it is just informational and not something that you may necessarily need unless you really want to go that root.  I suspect that the more affordable root for most people is simply use what you have. We all have USBs(cheap and very reliable for the most part if you buy brand names), and then you have your computer hard drives which are too very reliable to a point. Now if your operating system goes south/bad then restoration of the hard drive files will be next to impossible as it will have to be reformatted to re-install the OS.

  So perhaps the easiest thing for you to do is simply get a couple USBs(32gb Sandisk) format them on your keyboard to ensure the FAT32, label one as your live disk and do everything on it keeping only that one plugged into your keyboard(as Chris in many posts has suggested), then when you think that you have made/created significant updates to the live, use your computer to do a complete "CLONE"  which is a COPY to COPY everything at once to the second USB.  Of course once completed, plug the the clone in the keyboard and ensure that your changes and other Registrations are reflected/play correctly. You should also ensure that when removing any external drive/usb from your laptop that you "Safely remove hardware and eject media" from the computer, recommended procedure from Microsoft.  I have played with this and realistically for any FAT32 USBs, I have never had issues with just yanking them out of the computer.  However the new format exFAT has a number of times cause me issues.

  Note: adding a folder(only) to a cloned USB under the root directory(not creating a sub-folder that is) will not hinder your backup/recovery capabilities , just ensure that what ever folder you add it's added to live usb as that is always your backup source.

   Can't say anything more, my approach has one bottom line and that is that for me regardless of how complicated or  detailed my approach is, I have a backup that I can quickly restore from, it is simple, solid and proven with total file integrity.

              Regards.      dom 

[adrianed]

Dom,
I don't store items on windows drive I always keep windows separate except for programs which are also stored on the C drive
In my computers I have 3 separate hdd,s a windows and programs drive, an everything else drive and a 2 TB drive for only backups
Adrian

[dlepera]

Adrian,

You stated " read some questions and answers online this week on a computer forum a question was asked if it was a good idea to store items on ssd hdd or usb drives and most of the replies were a resounding no"

  From this statement we evolved to where now we have some feed back on basic methods than can be used to help ease our minds a little on how to safeguard and stored data with better integrity and recoverability. 

  Like every storage story there are always important thing that need to be considered. Things like, how much data are we talking about, how much are we willing to spend to ensure availability and recoverability, is this data for business or personal.

  Let's forget about reliability of storage media. One can read for days and at the end of it all you will learn what you already know.  There is no guarantee that any type of storage media will not fail so forget that fear as it will eventually happen. I have seen all kinds of storage hardware evolution, magnetic storage media to very expensive electronic storage media. In time they all failed and will continue to fail. 

The only thing to ease your mind and help build your confidence in storage media is how you plan redundancy of hardware and a backup strategy. This backup strategy is not only for the loss of data but think of how easy it is to corrupt your data. Believe me, with all my backups of Registration files, in my first few months with my SX700, I had corrupted many of them. But due to my backup strategy I was able to recover everything to a good point in time very quickly and easily. 

   So Bogdan uses NAS and there you have the ability to implement hardware redundancy, as well as a good backup strategy. He has to maintain his environment but it is resident and his data is secured. The initial cost of ownership is higher at first but eventually peters out to NIL over time as he owns "everything"

   Joe mentioned CLOUD and this is the way of the future. Locally stored data , music, phone content, computer content, etc will be storage on media somewhere in the world with our name on it and you will never have to think about media failure and backup as the provider will do it all for you. 

As nice as CLOUD is it too has it's downside.
1. You don't own the storage media, but you do pay for the providers cost of ownership, software upgrades and environmental maintenance reflected in your raising monthly fees.

2. You have no idea where in the world our data resides/stored. Should you care? No need too. I sure do!

3.  Is your stored data secured? The provider will tell you yes under your account and that your data is encrypted. Do you have the encryption key, you may, but who has the master key?  Not you!!

4. Over time you may just keep buying cloud storage not knowing how much your are using and really do you care? Perhaps not until one day you really want to bring back that data in house or change provider and you find out that you can't afford the resources to store it because you did not keep track of growth since a few extra dollars on monthly fee made that all transparent to you.  Try moving it to another provider. Good luck.

5. Can your data be compromised, sure can. Look around you. Social Media data is on the CLOUD.

   I leave you with this,  "what can we put our confidence in these days" ??          Only the ALMIGHTY!

    Best Regards.                  dom   



   

[JohnS (Ugawoga)]

Just get a decent 2 Terrabyte usb drive
Put all backups on that .
Mainly put a System image on it of C and D drives on a computer.
Never had any problems with usb storage .

Always update a System image every three or six months by re doing it and deleting the old one.
If you are paranoid back everything up and duplicate on another drive
Two 2 terabyte drives £220 approximately, but even that would save you hours of heartache.
I always system image with Windows 11 with no problems at all

Talking about degradation. I still have old usb sticks over 10years old working as good as new.