Since Xcode 11, we can add or remove Swift Package by GUI operation. Apple guide you in Xcode 11 Release Notes as follows:
How to use
Adding a Swift Package in Xcode is easy. Open your Xcode project, Click
File > Swift Packages > Add Package Dependency....
Enter the URL that provides the Swift Package.
Specify the version. It seems that you can also specify the branch and commit number.
Specifies to use the Swift Package to load to
Target. See the Add Target column in the figure below.
Loading is completed by the above operations. It can check the loaded Swift Package on the Info tab of the Xcode project. You can add or remove from there.