Search behavior very strange on - SX700 - GOOD NEWS today!

Amwilburn · April 14, 2024, 04:14:25 PM

Quote from: dlepera on April 09, 2024, 09:20:06 AM
I would think that the solutions for that if they really thought it out would have been to cap the size of the external storage. Example my flag ship Double Din Kenwood car stereo unit years back would not accept anything greater than 4gb. Anything connected greater than that it would not recognize it. Easily done. In fact Yamaha does do a control on the external storage device format as you. It only supports Fats32.

Well, yes:
mp3's on 4gb, at most that's ~1000 search entries for mp3's.

Fat 32 has a maximum 2TB; my own personal experience shows that if I filled a 2TB full of arranger files, it would probably contain in the order of 38 million files. (it doesn't scale directly from my 99gb because I wouldn't use 200 times the sample area, it's still the same expansion files regardless of how many styles and registrations I add)

My own stick only 30GB was small files like registrations, styles and midis, 62gb was expansion files. (the rest were Mp3's and wavs).

Even my PC struggles when I'm searching 10s of millions of files. And my cell phone search isn't that fast at 12 thousand files. We're asking an arranger to search many more times that; and if you use a cheaper USB stick (they all say USB3 compatible, but during a search it's the read speed that matters)

Mark

KurtAgain · April 14, 2024, 04:41:28 PM

Quote from: Amwilburn on April 14, 2024, 04:01:23 PM
[...] searches get exponentially slower with more search entries.

Well, that depends on the search algorithm used and the ability and willingness of the programmer.

Amwilburn · April 14, 2024, 05:05:01 PM

True.
*Brute force* searches, get exponentially slower, and I can't see anyone coding anything significantly more complex than a linear search for a keyboard, but hey, I haven't typed a single line of code in over 20 years, so I'm hardly up to date!

dlepera · April 14, 2024, 06:22:00 PM

Hi Mark,

After everything is all said and done, while speed is always key and on demand for anything, in our case I don't think anyone is really concerned about the speed of an arranger search. I have had to sit as long as 45 seconds for mine a few times before it returned a search result. I knew my particular search would take some time because I connected a monster USB. However, because my search was for something that I knew would get searched and found, I was still happy. The bottom line is not speed but capability/functionality.

This under normal conditions would not affect any performer or average player if they stick to bare minimum.

But as you stated

"Although I should point out, once we get out of our pack rat phase (if ever), and begin to streamline the styles/registrations/multipads that was *actually* use, the search system should work. But yes, they should have assumed more."

this I don't think will ever happen in our life time. Just the nature of the beast. So Yamaha has to just realize this and maybe even step up to the plate and allow at least owner management of file manipulation and open the flood gate to the fact that searches are going to be huge so write software to make that happen. Your PC does not fail a search because your storage is too large does it.

We talk about Samsung Cell phones. Yes they are fast, very fast, but in order to keep up with demands, and work with their competitors to ensure profitability is widely spread to everyone, they too are now being challenged I think. To make room for all these wonderful apps and blue tooth devices, new games, how to you see them keeping up with the speeds that we have been used to all these years?

Here is how this layman see it. Faster chips mean more $$$ to produce. Demand for new device technology connectivity has to be accommodated by the manufacture so to keep costs containable and still up profitability, the easiest way to do it as I see it, is to first make cell phones with anything over 128gb much more expensive to deter buyers and get them to just buy the 128gb cell phone. Then remove the Micro USB capability which puts a burden on additional lag time for searches and navigation. Now move that to the cloud which has a multitude of benefits for other technology companies like Google Cloud, BOX, Samsung Cloud etc as well. They now make money off you, but the key item here I think, is that any slowdown for a file access accountability, will be transferred to the cloud service provider and not the Samsung's phone. Yes they will also tell you that they needed more space for device antennas, better camera's etc. No need to invest on faster chips now. Everyone gets a piece of the pie except for us.

We can go on and on with this subject, but the technology strategy is not one we can change. Just like Beta, VHS, Super/8MM cassettes , Turn tables, CD, DVD, Blue Ray . we are like fish on a hook and they know how to manipulate us. Way of life.

Your info is very good.

Thanks. dom

alka · April 15, 2024, 07:36:54 AM

Hi, I went back Sx900 to 1.05 and update to 1.11 but the search function keeps finding old styles, voices, even from the old USB and expansion packs,

pjd · April 15, 2024, 11:58:33 AM

Both the SX700 and SX900 use an eMMC device for internal data storage (MTFC4GACAJON-1M). The data clock is 48MHz and the bus operates in 4-bit mode (a 4-bit nibble every clock cycle). So, the maximum transfer rate is 24MBytes/sec. Writing is slower because flash has slower write time.

eMMC is used instead of USB because the CPU chip -- AM4376BZDN100 at 1GHz -- has built-in eMMC interfaces. This keeps cost low.

Possibly this could have an effect on search speed because that's a fairly narrow pipe in terms of bandwidth.

Not enough caffeine yet to think this through... pj

dlepera · April 15, 2024, 12:05:45 PM

Hi Alka.

The first thing to do is wipe all files from your USER storage area. Then do a factory reset to clear anything left in there like your Packs. Power up and verify storage using your utility function in the menu and ensure storage area is clean. Then load in your Version 1.11. This will give you a clean starting point and then address the results. If you have that old version handy and care to share it with me, I would appreciate a copy of it so that I can play around.

PJ, your feedback is always appreciated. so hope that caffeine sets off all those sparks of great thoughts.

dom

dlepera · April 15, 2024, 07:34:53 PM

Hi Pj.

Did some searching on eMMC device for internal data storage (MTFC4GACAJON-1M) and you got me worried now. Is this the same memory in the TYROS and Genos? Are we confirming built-in obsolescence with the SX700 and SX900 of maybe 3-5 years?

I found that life expectancy is between three and five years and then I saw this:

"The typical lifespan of eMMC NAND flash memory in terms of read/write cycles is between 3,000 and 10,000 cycles. However, this can vary depending on a number of factors, including: The type of eMMC NAND flash memory used. The temperature at which the eMMC is operated."

This means that each flash cell can be written to and erased 3,000 times before it starts to fail???

Can you please shed some light on this find. Is the USER memory part of this and if so is every search or indexing added to the cycle? Tell me that I am in left field on this one please>>>>>

dom

pjd · April 16, 2024, 12:22:28 PM

Quote from: dlepera on April 15, 2024, 07:34:53 PM
I found that life expectancy is between three and five years and then I saw this:

"The typical lifespan of eMMC NAND flash memory in terms of read/write cycles is between 3,000 and 10,000 cycles. However, this can vary depending on a number of factors, including: The type of eMMC NAND flash memory used. The temperature at which the eMMC is operated."

This means that each flash cell can be written to and erased 3,000 times before it starts to fail???

Hi Dom --

Yes, this is true. It is also a complicated subject about a known phenomenon.

Short answer: eMMC is used everywhere including cellphones, automobiles, etc. All NAND flash share this behavior. Developers know this and both hardware/software use techniques to prevent early cell (block) failure. This doesn't keep me awake at night.

Long answer: That 3 to 5 year estimate is based upon assumptions about usage patterns, etc. Linux and other embedded OSes use wear-levelling algorithms and other techniques. At the hardware level, there are error correcting codes (ECC) and spares. If a defective block is identified, it is replaced by a spare. The 3,000 to 5,000 program/erase cycles are per-cell, but the problem is managed at the block level. Thanks to wear-levelling and replacement, the eMMC will slowly shrink in usable capacity. There are hardware registers that track use and degradation.

BTW, I've seen estimates as high as 10,000 p/e cycles. Again, it's all based on usage patterns (how often flash memory is written).

Note that the life issue is for program (write) / erase cycles, not reads. Yamaha should be (and probably are) building search results in SDRAM, everyday primary memory with no life-cycle issues. I expect Yamaha to write data to be be saved to eMMC once during power down. BTW, this is why a user should never remove power while the OS is saving data to eMMC flash memory!

One big take away for all users, your flash drive will fail someday, too. Always keep multiple copies of your data! Your time is too valuable to waste when flash drives are so cheap.

Hope this helps -- pj

https://sandsoftwaresound.net/nand-flash-data-retention/
https://www.cnx-software.com/2019/08/16/wear-estimation-emmc-flash-memory/

dlepera · April 16, 2024, 03:17:55 PM

Hi PJ.

Good to know. So my question then is "how long do I keep my SX700 seeing that I purchased it in 2020. I have no idea when those writes are going to take place unless there is something specific that triggers the "write", also , how would I know when the OS is saving data so that I do not turn the keyboard off. I have never seen a message indicating that.

After all this can i safely say that it does not pay to buy a used upgrade keyboard as I will not know what it's life expectancy will be. I don't see it fair to sell mine to anyone and then that poor owner ends up with what I was trying to avoid by selling.

So two options I see, 1) run mine to the ground part it or throw it away, then buy new
2) buy a new backup motherboard and keep until mine dies, then swap it.

Does this make sense?

dom

KurtAgain · April 16, 2024, 04:01:50 PM

Dom, I think you're worrying unnecessarily. The important point is

Quote from: pjd on April 16, 2024, 12:22:28 PM
If a defective block is identified, it is replaced by a spare. [...] Thanks to wear-levelling and replacement, the eMMC will slowly shrink in usable capacity.

And you probably won't even notice it.

pjd · April 16, 2024, 06:06:20 PM

Quote from: dlepera on April 16, 2024, 03:17:55 PM
So two options I see, 1) run mine to the ground part it or throw it away, then buy new
2) buy a new backup motherboard and keep until mine dies, then swap it.

Does this make sense?

Hi Dom --

I say, go out and have an espresso or two. Maybe a glass or two...

I honestly don't worry about this stuff and I keep keyboards 10+ years. I drive old cars, too.

Kurt made note of the most important point. The eMMC will not suddenly drop dead, but a bad block here or there will slowly decrease the pool of available blocks, that is, the available capacity will decrease somewhat.

C'est la vie. By the time anything is noticeable, you'll be sorry that you bought that spare motherboard. Save the money and splash out on a new ax someday like all the rest of us gear-obsessed Forum members.

As to powering off during a write, the internal Linux system will perform an orderly shutdown. Problems may occur when people rip the power cord out of the wall while the system is fully ON and active. The power switch is a command to enter the stand-by state. As to USB drives, one shouldn't remove them when a drive's busy light is flashing.

Frankly, I worry more about my heart... -- pj

dlepera · April 16, 2024, 08:25:08 PM

Gentlemen,

Thanks for the re-assurance. I am not the worrying type, never have been, but I do like to plan to avoid inconveniences with the ability to obtain a deal when the time arises. I too have old cars and never have had to worry about their robust electronics for those that have them. Just contemplating if I should replace my points and condenser with an electronic module or maybe avoid that possible headache? I also have lots of your favorite PJ, vacuum tubes for my town and country radios and treat them like gold.

Anyway, I thank you and everyone contributing to this post for your support and information which I do value and it has also provided the temptation for me to explore these beasts so will now be looking around for a tear down SX700/900 if I find one cheap for education purposes.

Meanwhile I will be waiting to see if Yamaha will get back to me on my Search issue and pass that info on to you.

Best regards.

dom

dlepera · April 28, 2024, 10:40:39 AM

Hello to all following this post.

I did promise to provide a status on how things are going with Yamaha. And the good news still continues.

Status:
1. Have not heard anything from Yamaha yet. Frankly don't think I will so I have kind of given up on them
and wait for a miracle response some day to happen.
2. I have received over 12 personal emails from people with the similar problem both for SX700 and
SX900(mostly SX900) and after a long time of not being able to use the search facility they are now
working like new again. I have asked that they update their experience on the forum if they want, so this
is their prerogative.
3. Some told me that their attempts following my suggestions in the post did not work while others were
working in no time. Further communication clarified one simple point that I may not have made clear and
that was to let the keyboard "index" after the factory reset and once the indexing message appears the
issue should be gone which it was.

Procedure Recap:

1. Clear everything in your USER memory, you can do item by item manually, delete your packs, but the
easiest is go into Menu2, Utility, Storage, select "USER" which will be highlighted and then select
"FORMAT" . Once done the user area will be clean.
NOTE: ensure that no external device(USBs) are connected
2. I recommend a Reboot at this time.
3. Do the factory reset by going into Menu2, Utility, this time select "Factory Restore/Backup". Check all 6
items System, User Effect, Favorite, MIDI Registration, Live Control to get things as clean as possible.
Then selected "Factory Reset". This will be quick.
4. Now Reboot the keyboard. Understand that Reboot means turn off and on the keyboard by the way.
5. Give a couple minutes and then power it "Off"
6. Now re-install your your latest firmware version V1.11 following your Yamaha's instructions. See link.
https://europe.yamaha.com/en/support/updates/firm_psr-sx900.html
7. Once installed, remove your USB, ensure there is USB connected at this point.
8. Reboot the keyboard and do nothing. Just wait as long as possible or until the "Indexing" message show
up. I actually have never seen that message on my keyboard in all the years that I have owned it nor
have I seen it during my testing. However a few of you have told me that it does pop up. And some forum
post responses seem to indicate 15 minutes or more.
9. Once indexed with no USB connected, the magic seems to happen.
10. Reboot for the final time and your issue is gone.
11. Connect your USB and do your searches of different things. Styles, Regits, Voices, Pads, etc

This procedure is my own. I am not going to question why this sequence works this way. Maybe we can cut some step out, but since this is working not only for me but for others, I would stick with it.

Not:

Some have hit the brick wall file limit message after all this was done and thought that the issue was back. Well it was not back at all in there case. If the issue is back, this message will appear every single time you search. What you need to understand is that depending on the nature of your search criteria and the number of entries that are found that meet that criteria only the first 100 entries can only be put on the list as the memory list size can only hold 10 pages of 10 entries and the it stops writing the entry list. Hence you message. Be more specific with your list if you can and things will be displayed in your favor.

Hope this clarifies things.

New Observations:

I have not given up on my trying to determine "root cause" for this search issue so still playing around and testing things.

1. Going back to my records before the issue I was seeing those ".-" files that have been talked about in this
forum. I still have no clear understanding of them, but that is fine as I no longer have any of them around
any more. However I did noticed that as I was collecting styles from the forum and adding them to my
USBs, the list in my search keep showing an increase in the number of those entries showing up in my list
until the search issue appeared.

2. Not sure if it has any relevance. But I will say this. In all my testing that I performed and talked about in
this post, I was able to "break" my search ability every time.

3. Once I deleted them and in particular ".-E*", I no long can "break" my search. So could there be some
relationship to the way the keyboard handles them?

?? These were the first files that were listed in my
failing search and never got to see if any others would make my keyboard react the same way as I just
blew everything with ".-*" off my computer and USBs.

4. The path on my USB before cleanup was "Yamaha new styles/download/dec2021/usb drive/usb 2015/vse
ps3.Balken". I am trying to zero in on those and will try to locate them on the forum style sections so
that I can isolate them this time and do some further testing of the keyboard's behavior with them.

My curiosity has gotten the best of me so I will continue to play around with this issue until I reach my saturation point on this subject. I have asked if anyone has a copy of a previous firmware upgrade, something like V1.10 or prior and no one has sent me one yet. Please check you files and see if you have one lying around.. I reallllllllllly would like to play with that version, even if I can't break my search any longer At the moment. However I suspect that those ".-E*" may still be in a style folder on the forum so will hunt those down again. And yes the guy with backups of backups should have a copy but I am also very good at cleanup. This time too good.

Best regards! dom

johan · April 28, 2024, 02:16:50 PM

Hi dom,

Here a link to SX700 version 1.04 (link valid for one week): https://we.tl/t-OgsjwBCVa8
I also had problems with the search on my SX700 but then I have reduced the number of files on the USB and the issue seems to have disappeared now. But it is good to know that there is a more firm solution in case the issue comes back.

Johan

dlepera · April 28, 2024, 02:43:29 PM

Hello Johan.

Thanks for the link.. This is great as I now have something to work with different from the existing V1.11.
Btw, the search issue comes in two flavor.

One is the hard search where the search list will never change on you regards of what you search unless you follow my procedure, and the other is the search hitting the max entry limit which comes form tooooo many file entries with the similar name in your search criteria. If you get the second, which I think you got, then what you did is the way to go if your folders have redundancy and this becomes a pain for you.

For me I still have over 25,000 entries on my USB and growing(I am a bad boy this way) but I can search what I want by just being more selective in my search and the keyboard is still happy. Why do I do this one may ask. It's because I want to have everything I need all in one place so I don't have to keep switching USBs when I want something. There are many styles for a song in different folder and from different forum members so rather than to move all of them into one folder I just keep things in existing folders. My silly mentality may go against the flow of many recommendations in the posts of this forum, but file management for me is not an issue. I got burnt with the registrations once which was a lesson learned the hard way but now I use caution and have been good.

I do respect the limit that is talked about in this forum about ensuring that the number of registrations in a folder does not exceed the Yamaha recommended max threshold, I think I read 255. Regardless I have kept my down lower and just created multiple Registration folders.

I think you know not to move style files around if the registrations are using them so be careful when you reduce your folders and entries on the USB else as you know the registrations may not work.

All the best!

dom

Aquilauno · May 01, 2024, 05:19:59 AM

Hi, I've been busy these days, but today I was able to dedicate more time to my SX900. I confirm that the procedure by Dom, step by step, resolve the problem about the search funsion (with soft reset performed several times). Delete everything from user memory including expansion packs. Soft reset again, Keyboard left without USB for a couple of hours, firmware 1.11 installed again. Inserting the USB with styles (134,000 styles on board carefully divided) (don't ask me why. I have the same answer as Dom lol). To Wait the index again.
Indexing performed and search function restored and working. I'have is one problem, however, when entering the expansion menu (button on the right of the keyboard) and carrying out a search for voices, the problem is still present for the vce files, results are given that are no longer present in USB. The search for styles is perfect, the search for voices still presents a problem, both from the expansions/user, buttom in the right, both from display selected R1 R2 and R3 voice and using search funsion. All link shown in search works for the styles but not all for vce. Dom checks whether the search starting from the expansions menu (expansion/user button in the right, see photo B) works correctly. For my keyboard no work good. Now not have time for to study this, I will try as soon as possible to repeit the procedure with the reinstallation of the 1.11 firmware. Pietro

[attachment deleted by admin]

dlepera · May 01, 2024, 09:31:06 AM

Hello Pietro.

Very interesting. I have just performed the following search on my keyboard and here are my resulsts.

1. My expansion area shows Africa, Europe, Latin America & Oriental so that we are on the same page
2. I did a search for the letter "a", then "c", then "b" and "549"
3. All my searches in point #2 show.."The number of search results has exceeded the limit" of course - normal
- I did get different results of course with 10 pages of them and all different every time.
4. Now I do a selective search like "altair"
5. My result is only 1 page now with 1- Altair, 4-altair 494 and 4-altair 549
6. Did another search on "growlsax" and result was 4 pages, 3 full pages and last 4th with only 4 entries in it
7. Now with the Expansion button light on I selected my USB1, cleared my previous entries from point #6
and did a search on "acordian". Result was an empty list which it should be. I spelt accordion wrong
8. Did a search on my USB1 for "awtar" and got 3-Awtar 1G3 and 4-Awtar 621, so USB search working too.

This confirms that Mine is working fine and anyone can confirm my results to keep me honest.
The only thing I can suggest is what you are planning and that is to redo, after ensuring everything in User is clean. Should show 1gb available space. I think the memory size is the same for SX900 as it is for my SX700. Then do your thing and this time let it index again but twice.. Power off and on, index, power off and on a second time and let it index again ensuring that there is NO usb connected during that time. Then check things out. If that does not work, I will be at a loss.

As I mentioned I have no idea on the software logic so my workaround is only based on speculation. I have been trying to break my search function and I can't. It is still working great.

I have not had time yet to install the old version firmware provided to me to see if I can break things, but I still point my finger on those ".-E*" files as being my cause which broke things every single time when I had them on my USB with my styles. Once I cleaned them up from everywhere, I am unable to break this search function.

I will say one thing, though that I am not the only that has large number of files on a USB and good to see that it is also working on the SX900-why should it not? Just cumbersome for many unless you know how to organize your folder and do selective searches so that you do not hit the limit brick wall. An that message does not break anything as we have been seeing.

I would love to put my finger on the exact trigger that breaks the search function. So the more we communicate the better more we understand. I don't expect Yamaha to ever fix this issue so don't hold your breath for that one. However, I do believe in miracles!

Best Regards! dom

mikevines · May 02, 2024, 02:05:07 AM

The vagueness on recommended number of files in a directory I think comes down to the internals of the FAT32 format. A directory can have 65536 'entries'. Each file will require multiple 'entry' - at least 2, and more if longer file names are used. I think therefore the 255 guide is not unreasonable, but not absolute. I guess once you exceed the limit, the addition of additional files to the directory would fail.

Mike

Aquilauno · May 02, 2024, 09:59:22 AM

I hope to have time to try again soon. However, one positive thing is that the USB search for styles works well now, the results are correct and no false links with files not present on the USB have recurred. This is a big step for me. Consider that it was on the second attempt. The first one I didn't follow Dom's instructions to the letter. The advice to avoid style names with "strange" letters is excellent. Fat32 formatting should affect file size and not the number of files, but I don't have the knowledge to say this for sure. What is important is that the path indicated by Dominic is the right one to solve the problem of the search not updating correctly. I confirm that it works... great job Dom.
PS I realized that some Voices that I had modified and saved in user, they remained in user memory. I didn't notice it before because I entered the menu from styles and therefore deleted what I saw (but not the voices, which I couldn't see from there). Today, entering from the expansion/user button on the right, I went to user and found about ten voices stored and not deleted, "my fault". All that remains is to repeat Dom's procedure.

bhergar56 · May 02, 2024, 03:49:43 PM

Quote from: BogdanH on April 06, 2024, 10:42:02 AM
hi dom
PSR-SX900/700 service manual is circulating on internet and if you're interested I can send it to you. But to benefit from it, certain electronic knowledge is required, though.

As can be seen on Youtube, disassembling and assembling the keyboard is relative simple. I did that (greased the keybed) and I had no problems at all. However, if you don't feel confident, then just don't do it.
Anyway, your issue is purely software based and so there's no point to open the keyboard (looking inside won't help).

What amazes me about Yamaha arrangers is the fact, that "full factory reset" (as from the box state) simply doesn't exist. That would obviously solve your problem (I say "your", because I don't use search function at all).

Bogdan

Hello Bogdan, reading your post, could you plese facilitate to me a copy of the psr-sx 900/700 service manual? I appreciate very much your help.

Best Regards

Bernardo.

dlepera · May 08, 2024, 04:51:19 PM

Hi All.

As promised, I did say that I would provide a status of this issue once Yamaha touched base with me.

Well, I have to say that today I had a very pleasant phone conversation with the Technology Product Manager in Canada and he has assured me that Yamaha is very interested in replicating and understanding the details associated with this issue.

In fact CS Group Quality Assurance Department Musical Instruments Business Unit Yamaha Corporation of Japan is involved and seeking basically answers to questions that we have all documented in this post. I provided the link to this post for their documentation gathering and analysis.

One of their ask was to help them identify how to break the search that I was sooo good at doing before. Now, (as previously stated) I kind of kick myself in the butt that I deleted those ".-E*" 2005 Balken styles without taking a backup for this particular reason as it is more than obvious that once they were deleted off all my USBs, at least on my keyboard, immediately I was unable to break the search again. Not sure if I mentioned, I have to assume that they were created on a Tyros 1 or 2 judging from the TY suffix and the release dates of 2002 -2005?

So at least for now, we can give Yamaha credit for taking interest!! Thanks You Yamaha Canada and Japan Corp.

Regards dom.

dlepera · May 09, 2024, 02:01:53 PM

Hi .

I have opened a separate Post related to the Search behavior which is really a continuance of this but more details are provided with a very favorable finding.

https://www.psrtutorial.com/forum/index.php/topic,69438.msg522537.html#msg522537

Regards dom.