Number of files that Chains can handle

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Number of files that Chains can handle

Stevethefiddle
This came up on the forum recently and I don't know the answers:

1) How many files can be selected in one go for batch processing with a Chain?

2) What is the limiting factor?

3) What actually happens if the user selects too many files?

4) Is it a bug that there's a limit, or a bug that we don't tell the
user when they have exceeded the limit?

Steve

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Audacity-quality mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/audacity-quality
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Number of files that Chains can handle

James Crook
I don't see any obvious limit in our code, in that we are using
wxArrayString which supposedly grows as big as needed.
--James.

On 1/19/2017 4:38 PM, Steve the Fiddle wrote:

> This came up on the forum recently and I don't know the answers:
>
> 1) How many files can be selected in one go for batch processing with a Chain?
>
> 2) What is the limiting factor?
>
> 3) What actually happens if the user selects too many files?
>
> 4) Is it a bug that there's a limit, or a bug that we don't tell the
> user when they have exceeded the limit?
>
> Steve


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Audacity-quality mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/audacity-quality
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Number of files that Chains can handle

Stevethefiddle
My computer is not up to the job of testing this, but evidence from
users suggests that there is a limit of around 2000 files, though I've
not seen detailed (or even useful) reports of what actually happens if
the "limit" is exceeded, other than "doesn't work".

Steve

On 19 January 2017 at 18:05, James Crook <[hidden email]> wrote:

> I don't see any obvious limit in our code, in that we are using
> wxArrayString which supposedly grows as big as needed.
> --James.
>
> On 1/19/2017 4:38 PM, Steve the Fiddle wrote:
>> This came up on the forum recently and I don't know the answers:
>>
>> 1) How many files can be selected in one go for batch processing with a Chain?
>>
>> 2) What is the limiting factor?
>>
>> 3) What actually happens if the user selects too many files?
>>
>> 4) Is it a bug that there's a limit, or a bug that we don't tell the
>> user when they have exceeded the limit?
>>
>> Steve
>
>
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> _______________________________________________
> Audacity-quality mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/audacity-quality

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Audacity-quality mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/audacity-quality
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Number of files that Chains can handle

Gale
Administrator
In reply to this post by James Crook
>From what has been reported several times, the file name box in
"Select file(s) for batch processing..." doesn't show the names of
all the selected files, then when processing, the file name gets
corrupted, as in this image:
http://i.imgur.com/088wN.png .

The same thing is reported in the recent Forum topic that I guess
Steve refers to:
http://forum.audacityteam.org/viewtopic.php?p=320653#p320653

It was not apparent until later on that the user was trying to process
over 4000 files.

The limit on number of files has been reported on Windows between
800 and 2000. The limit does seem to depend on the length of the
file names, according to reports.

There also used to be an error processing the AUTOSAVE file, but
AFAIK that does not happen now after its conversion to binary.

Is it a bug or enh? Probably enh?


Gale



On 19 January 2017 at 18:05, James Crook <[hidden email]> wrote:

> I don't see any obvious limit in our code, in that we are using
> wxArrayString which supposedly grows as big as needed.
> --James.
>
> On 1/19/2017 4:38 PM, Steve the Fiddle wrote:
>> This came up on the forum recently and I don't know the answers:
>>
>> 1) How many files can be selected in one go for batch processing with a Chain?
>>
>> 2) What is the limiting factor?
>>
>> 3) What actually happens if the user selects too many files?
>>
>> 4) Is it a bug that there's a limit, or a bug that we don't tell the
>> user when they have exceeded the limit?
>>
>> Steve
>
>
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> _______________________________________________
> Audacity-quality mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/audacity-quality

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Audacity-quality mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/audacity-quality
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Number of files that Chains can handle

James Crook
The limit could easily be in Windows/WxWidgets code rather than in our code.
It would be no surprise to me if windows had a limit of 64K for the
total length of all the file name strings.

We know windows XP and 7 get very slow with 'large' directories. It's a
reason we use subdirectories for our data files.

A proper fix could well entail writing our own independent FileDialog
that is not derived from the one in wxWidgets.  It does not seem worth
it to me - work involved high, relative to small benefit to the users
affected by the problem.

Possibly useful is to detect > 1000 files selected (which we may not be
able to do reliably!), and direct the user to an appropriate help page.  
We could also allow directories to be selected, which would provide a
workaround for most users with large numbers of files, with some work
for us to do, but less than a FileDialog rewrite.

--James.


On 1/19/2017 6:54 PM, Gale Andrews wrote:

> >From what has been reported several times, the file name box in
> "Select file(s) for batch processing..." doesn't show the names of
> all the selected files, then when processing, the file name gets
> corrupted, as in this image:
> http://i.imgur.com/088wN.png .
>
> The same thing is reported in the recent Forum topic that I guess
> Steve refers to:
> http://forum.audacityteam.org/viewtopic.php?p=320653#p320653
>
> It was not apparent until later on that the user was trying to process
> over 4000 files.
>
> The limit on number of files has been reported on Windows between
> 800 and 2000. The limit does seem to depend on the length of the
> file names, according to reports.
>
> There also used to be an error processing the AUTOSAVE file, but
> AFAIK that does not happen now after its conversion to binary.
>
> Is it a bug or enh? Probably enh?
>
>
> Gale
>
>
>
> On 19 January 2017 at 18:05, James Crook <[hidden email]> wrote:
>> I don't see any obvious limit in our code, in that we are using
>> wxArrayString which supposedly grows as big as needed.
>> --James.
>>
>> On 1/19/2017 4:38 PM, Steve the Fiddle wrote:
>>> This came up on the forum recently and I don't know the answers:
>>>
>>> 1) How many files can be selected in one go for batch processing with a Chain?
>>>
>>> 2) What is the limiting factor?
>>>
>>> 3) What actually happens if the user selects too many files?
>>>
>>> 4) Is it a bug that there's a limit, or a bug that we don't tell the
>>> user when they have exceeded the limit?
>>>
>>> Steve
>>
>> ------------------------------------------------------------------------------
>> Check out the vibrant tech community on one of the world's most
>> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
>> _______________________________________________
>> Audacity-quality mailing list
>> [hidden email]
>> https://lists.sourceforge.net/lists/listinfo/audacity-quality
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> _______________________________________________
> Audacity-quality mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/audacity-quality
>
>


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Audacity-quality mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/audacity-quality
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Number of files that Chains can handle

Stevethefiddle
Thanks for the additional information Gale.
Imo it's definitely a "bug", though probably not really "our" bug.

On 19 January 2017 at 19:21, James Crook <[hidden email]> wrote:

> The limit could easily be in Windows/WxWidgets code rather than in our code.
> It would be no surprise to me if windows had a limit of 64K for the
> total length of all the file name strings.
>
> We know windows XP and 7 get very slow with 'large' directories. It's a
> reason we use subdirectories for our data files.
>
> A proper fix could well entail writing our own independent FileDialog
> that is not derived from the one in wxWidgets.  It does not seem worth
> it to me - work involved high, relative to small benefit to the users
> affected by the problem.

In the light of your next comments, +1

>
> Possibly useful is to detect > 1000 files selected (which we may not be
> able to do reliably!), and direct the user to an appropriate help page.

Stopping the user from selecting too many files would be a big
improvement if we can do it reliably. I don't think that we need to
take it to the edge - most users only want to process a handful of
files, it's only very occasionally that we hear of users wanting to
process thousands.

So we're working on the assumption that it is the number of 'file
names' rather than the number of 'files'?

If evidence suggests that 500 files is a safe number, then I think
that would be a reasonable limit, (and it is the number that our last
user with this problem chose for himself). Gale has suggested that
over 800 could be a problem.

> We could also allow directories to be selected, which would provide a
> workaround for most users with large numbers of files, with some work
> for us to do, but less than a FileDialog rewrite.

I think that would be useful in itself.

It has also been requested that there's a recursive option (all files
in the selected folder and sub-directories). That would be more work,
but greater benefit for some users.

Steve

>
> --James.
>
>
> On 1/19/2017 6:54 PM, Gale Andrews wrote:
>> >From what has been reported several times, the file name box in
>> "Select file(s) for batch processing..." doesn't show the names of
>> all the selected files, then when processing, the file name gets
>> corrupted, as in this image:
>> http://i.imgur.com/088wN.png .
>>
>> The same thing is reported in the recent Forum topic that I guess
>> Steve refers to:
>> http://forum.audacityteam.org/viewtopic.php?p=320653#p320653
>>
>> It was not apparent until later on that the user was trying to process
>> over 4000 files.
>>
>> The limit on number of files has been reported on Windows between
>> 800 and 2000. The limit does seem to depend on the length of the
>> file names, according to reports.
>>
>> There also used to be an error processing the AUTOSAVE file, but
>> AFAIK that does not happen now after its conversion to binary.
>>
>> Is it a bug or enh? Probably enh?
>>
>>
>> Gale
>>
>>
>>
>> On 19 January 2017 at 18:05, James Crook <[hidden email]> wrote:
>>> I don't see any obvious limit in our code, in that we are using
>>> wxArrayString which supposedly grows as big as needed.
>>> --James.
>>>
>>> On 1/19/2017 4:38 PM, Steve the Fiddle wrote:
>>>> This came up on the forum recently and I don't know the answers:
>>>>
>>>> 1) How many files can be selected in one go for batch processing with a Chain?
>>>>
>>>> 2) What is the limiting factor?
>>>>
>>>> 3) What actually happens if the user selects too many files?
>>>>
>>>> 4) Is it a bug that there's a limit, or a bug that we don't tell the
>>>> user when they have exceeded the limit?
>>>>
>>>> Steve
>>>
>>> ------------------------------------------------------------------------------
>>> Check out the vibrant tech community on one of the world's most
>>> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
>>> _______________________________________________
>>> Audacity-quality mailing list
>>> [hidden email]
>>> https://lists.sourceforge.net/lists/listinfo/audacity-quality
>> ------------------------------------------------------------------------------
>> Check out the vibrant tech community on one of the world's most
>> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
>> _______________________________________________
>> Audacity-quality mailing list
>> [hidden email]
>> https://lists.sourceforge.net/lists/listinfo/audacity-quality
>>
>>
>
>
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> _______________________________________________
> Audacity-quality mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/audacity-quality

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Audacity-quality mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/audacity-quality
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Number of files that Chains can handle

Gale
Administrator
On 19 January 2017 at 19:48, Steve the Fiddle <[hidden email]> wrote:

> Thanks for the additional information Gale.
> Imo it's definitely a "bug", though probably not really "our" bug.
>
> On 19 January 2017 at 19:21, James Crook <[hidden email]> wrote:
>> The limit could easily be in Windows/WxWidgets code rather than in our code.
>> It would be no surprise to me if windows had a limit of 64K for the
>> total length of all the file name strings.
>>
>> We know windows XP and 7 get very slow with 'large' directories. It's a
>> reason we use subdirectories for our data files.
>>
>> A proper fix could well entail writing our own independent FileDialog
>> that is not derived from the one in wxWidgets.  It does not seem worth
>> it to me - work involved high, relative to small benefit to the users
>> affected by the problem.

I think there could be more potential benefits assuming they require a
custom dialog. Examples might be a file play preview, a search box
for Windows and Linux (assuming none of the common window
managers on Linux have that), or a slide-out for the metadata window.

If the only benefits of a custom dialogue were to allow processing
of more files in Chains, I agree it is probably not worth it for that.


> In the light of your next comments, +1

The suggestions to allow users to select a directory, and to search
recursively in it would have their own merit in any case.


>> Possibly useful is to detect > 1000 files selected (which we may not be
>> able to do reliably!), and direct the user to an appropriate help page.
>
> Stopping the user from selecting too many files would be a big
> improvement if we can do it reliably. I don't think that we need to
> take it to the edge - most users only want to process a handful of
> files, it's only very occasionally that we hear of users wanting to
> process thousands.
>
> So we're working on the assumption that it is the number of 'file
> names' rather than the number of 'files'?

I think the limit "could" be the total number of characters in the
strings containing the file names.


> If evidence suggests that 500 files is a safe number, then I think
> that would be a reasonable limit, (and it is the number that our last
> user with this problem chose for himself). Gale has suggested that
> over 800 could be a problem.
>
>> We could also allow directories to be selected, which would provide a
>> workaround for most users with large numbers of files, with some work
>> for us to do, but less than a FileDialog rewrite.
>
> I think that would be useful in itself.
>
> It has also been requested that there's a recursive option (all files
> in the selected folder and sub-directories). That would be more work,
> but greater benefit for some users.
>
> Steve

Another idea might be to allow users to select one or more LOF files
that listed the files to be processed. Few users would have such
files, but perhaps Audacity could generate a LOF file from multiple
selected files, if the user selected "too many" files?


Gale


>> On 1/19/2017 6:54 PM, Gale Andrews wrote:
>>> >From what has been reported several times, the file name box in
>>> "Select file(s) for batch processing..." doesn't show the names of
>>> all the selected files, then when processing, the file name gets
>>> corrupted, as in this image:
>>> http://i.imgur.com/088wN.png .
>>>
>>> The same thing is reported in the recent Forum topic that I guess
>>> Steve refers to:
>>> http://forum.audacityteam.org/viewtopic.php?p=320653#p320653
>>>
>>> It was not apparent until later on that the user was trying to process
>>> over 4000 files.
>>>
>>> The limit on number of files has been reported on Windows between
>>> 800 and 2000. The limit does seem to depend on the length of the
>>> file names, according to reports.
>>>
>>> There also used to be an error processing the AUTOSAVE file, but
>>> AFAIK that does not happen now after its conversion to binary.
>>>
>>> Is it a bug or enh? Probably enh?
>>>
>>>
>>> Gale
>>>
>>>
>>>
>>> On 19 January 2017 at 18:05, James Crook <[hidden email]> wrote:
>>>> I don't see any obvious limit in our code, in that we are using
>>>> wxArrayString which supposedly grows as big as needed.
>>>> --James.
>>>>
>>>> On 1/19/2017 4:38 PM, Steve the Fiddle wrote:
>>>>> This came up on the forum recently and I don't know the answers:
>>>>>
>>>>> 1) How many files can be selected in one go for batch processing with a Chain?
>>>>>
>>>>> 2) What is the limiting factor?
>>>>>
>>>>> 3) What actually happens if the user selects too many files?
>>>>>
>>>>> 4) Is it a bug that there's a limit, or a bug that we don't tell the
>>>>> user when they have exceeded the limit?
>>>>>
>>>>> Steve
>>>>
>>>> ------------------------------------------------------------------------------
>>>> Check out the vibrant tech community on one of the world's most
>>>> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
>>>> _______________________________________________
>>>> Audacity-quality mailing list
>>>> [hidden email]
>>>> https://lists.sourceforge.net/lists/listinfo/audacity-quality
>>> ------------------------------------------------------------------------------
>>> Check out the vibrant tech community on one of the world's most
>>> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
>>> _______________________________________________
>>> Audacity-quality mailing list
>>> [hidden email]
>>> https://lists.sourceforge.net/lists/listinfo/audacity-quality
>>>
>>>
>>
>>
>> ------------------------------------------------------------------------------
>> Check out the vibrant tech community on one of the world's most
>> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
>> _______________________________________________
>> Audacity-quality mailing list
>> [hidden email]
>> https://lists.sourceforge.net/lists/listinfo/audacity-quality
>
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> _______________________________________________
> Audacity-quality mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/audacity-quality

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Audacity-quality mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/audacity-quality
Loading...