Fix undefined behaviour in get_num_outputs in CoreAudio driver (#594)

See discussion in #591 for details. Basically an incorrect size was
being allocated for the CoreAudio buffer list for a device. It was being
allocated by a VLA (which already did not quite fit the semantics of the
list) and the length calculated could be 0 (instead of the size of the
struct with no buffers elements) causing undefined behaviour.

This corrects it to allocate the amount of memory required by the
CoreAudio framework function and adds a check for the size retrieval and
for the dynamic allocation. This change passed UBSan in my test where
before the change it did not.

Fixes #591
This commit is contained in:
Foster McLane 2019-11-13 15:13:54 -05:00 committed by Tom M
parent bebb5571ae
commit 19eacc6b60

View file

@ -83,10 +83,15 @@ get_num_outputs(AudioDeviceID deviceID)
pa.mScope = kAudioDevicePropertyScopeOutput;
pa.mElement = kAudioObjectPropertyElementMaster;
if(OK(AudioObjectGetPropertyDataSize(deviceID, &pa, 0, 0, &size)))
if(OK(AudioObjectGetPropertyDataSize(deviceID, &pa, 0, 0, &size)) && size > 0)
{
int num = size / (int) sizeof(AudioBufferList);
AudioBufferList bufList[num];
AudioBufferList *bufList = FLUID_MALLOC(size);
if(bufList == NULL)
{
FLUID_LOG(FLUID_ERR, "Out of memory");
return 0;
}
if(OK(AudioObjectGetPropertyData(deviceID, &pa, 0, 0, &size, bufList)))
{
@ -98,6 +103,8 @@ get_num_outputs(AudioDeviceID deviceID)
total += b.mNumberChannels;
}
}
FLUID_FREE(bufList);
}
return total;