The answer has been reworded.
At first I would say that number 3 is not a liar. It is because liars are usually smart enough to not to accuse anybody, if they are not sure they can make the words probable. If 3 did it, he wouldn't say "4 is lying" because he doesn't know what he is going to say. He does not know if number 4 has an alibi. It doesn't matter if you ask them one by one (whether they know what others say or not), or as a group. Normally 3 would say "I didn't do it", "4 did it", "I don't know, but I saw 4 playing ball near your window 5 minutes earlier" etc.
Number 4 says "I swear I didn't". A liar always needs to ensure his accuser, he's always trying to raise some emotions. This is why he uses big words like "I swear", "I vow", "I ensure" etc. He should have said "3, are you stupid? I wasn't there! Why do you lie?" or if he hadn't heard what 3 said just "I didn't" without "I swear".
Probably number 3 says "4 is lying" because he knows him well. 4 lies very often so 3 is trying to say to you "do not believe whatever 4 says, I know him, if you ask him, he will lie." Maybe 3 has the same experience as you? Maybe 3 asked 4 for something and was deceived? He trusts him no more and he's giving an advice to you.
Why don't you say there could be two or more liars? Why don't you think there were two perpetrators? For example 1 and 2: 2 says nothing about his guilt. He doesn't even say "I didn't". You ask him "did you?" and he says "1 is telling the truth". Strange for me.
At first I agreed with the other answer (I've even up-voted it), but this is not as clear as it seems for the first sight. I only wanted to show that 3 should not be taken as responsible, because this is not a puzzle, there is some fuzzy-logic in here.
(Please excuse me the first version of this answer)