Abstract: Multi-channel speech separation using speaker’s directional information has demonstrated significant gains over blind separation. However, it has two limitations. First, substantial performance degradation is observed when the coming directions of two sounds are close. Second, the result highly relies on the precise estimation of the speaker’s direction. To overcome these issues, this paper proposed 3D features and anassociated 3D neural beamformer for speech separation. Previous works in this area are extended in two important directions. First, the traditional 1D directional beam patterns are generalized to 3D. This enables to extract speech in any target region in the 3D space. Thus, speakers in the same direction but with different elevations or distances become separable. Second, to handle the speaker location uncertainty, previously proposed spatial feature is extended to a new 3D region feature. The proposed feature and model are evaluated under an in-car scenario. Experimental results demonstrated that the proposed 3D region feature and 3D beamformer can achieve comparable performance to that with ground truth speaker location input.

Scenario

In order to evaluate our proposed method in real-life applications, we consider a in-car scenario (Figure 1). As shown in Figure 2, there are 4 potential speakers and corresponding regions in a car: the main driver (S1), the co-driver (S2) and two passengers (S3 & S4) sitting in the back. We take the main driver's voice as the target. It can be seen from the top view that the azimuths of the main driver (S1) and the passenger in the back seat (S3) are very close. In this case, it is difficult to distinguish these two speakers with the spatial feature only based on azimuth.

Spatial Features

We assume that the main driver (as well as other passengers) will be located within a limited region (a 3D box as shown in the figure below, each dot indicates a potential speaker position). The length, width, height and corresponding distance to the microphone array center of this box, are measured according to the specific car. With this information, we can calcalate the expectation center point of the 3D box, and consider the center point as the speaker location.

1D Spatial feature: The spatial features based on azimuth that fed into the separation network are always extracted using the fixed center locations of 4 regions (boxes). As shown in the figure, the center locations of four regions are marked with the cross. Only the azimuths of the center points are used to calculate the spatial features.

3D Spatial feature: The fixed center locations (including azimuth, elevation and distance) of 4 regions are used to calculate 3D spatial features.

3D Region feature: We can also sample 8 vertexes of the 3D box to obtain a full spatial view of the whole region. As shown in the figure, 4 (regions) × 9 (spatial positions) spatial features that computed from azimuth, elevation and distance will be extracted and input to the separation model. We design a simple attention mechanism (i.e., weight-and-sum) to make the model selectively focus on useful spatial positions.